paper/NeurIPS_2026__AffectGPT_RL.pdf 
| **Open-Vocabulary Multimodal Emotion Recognition (OV-MER)** removes the restriction of a predefined label space, enabling fine-grained emotion understanding. Instead of relying on discriminative classifiers, OV-MER uses generative models to capture the full spectrum of emotions, and evaluates them with **emotion wheels (EWs)**. Existing approaches are trained with a *token-level* loss, which is **misaligned** with the metrics actually used in OV-MER: *joyful* and *awful* share high token-level similarity yet are semantically far apart. Worse, EW-based metrics are **non-differentiable** โ they involve stochastic sampling, discrete word-to-word clustering, and set-cardinality operations โ so they cannot be optimized by gradient backpropagation at all. **AffectGPT-RL** addresses this with reinforcement learning, which can optimize non-differentiable objectives. EW-based metrics are used as **reward functions**, so that the training objective is directly aligned with the evaluation metric. Extensive experiments reveal the role of reinforcement learning in this task. |
| **(Framework)** This is the first work to reveal the role of reinforcement learning in OV-MER. **(Experiment)** We conduct systematic experiments, revealing the role of the reasoning process, the impact of different rewards, and the generalization of reinforcement learning to other tasks. **(Performance)** Our method achieves state-of-the-art performance on fine-grained and basic emotion recognition, highlighting the potential of reinforcement learning in affective computing. |

| **Pipeline.** Training has two phases. In the *cold-start* phase we use a large-scale, coarse-grained dataset to build emotion understanding and output-format alignment. In the *RL* phase we use a high-quality, fine-grained dataset for reward calculation and policy updates. [run-release.sh](run-release.sh) collects the whole release as **five commands in four stages**: **Stage 0** sets up the environment and prepares the data, **Stage 1** (cold start) and **Stage 2** (RL) each run a training instance followed by an inference instance, and **Stage 3** computes the emotion-wheel score. The **[Reproduction](#-reproduction)** section below documents exactly these instances. |
| Validated environment โ every command in [run-release.sh](run-release.sh) was run in the conda env **`vllm3`** (we have not tested other envs): * python == 3.10.15 * pytorch == 2.8.0+cu129 (CUDA runtime 12.9), on a single NVIDIA L40 (48 GB) * vllm == 0.11.0 * transformers == 4.53.2 * tokenizers == 0.21.4 * trl == 0.19.1 The RL models import `trl`, and [statistic.py](statistic.py) imports `vllm` to turn the generated sentences into open-vocabulary labels, so both go in alongside the regular AffectGPT dependencies. Nothing has to be compiled: `vllm` ships its own attention kernels, so there is no `flash-attn` build step. |
| **[Environment Preparation]** All paths are hard-coded in [config.py](config.py). `REPO_ROOT` is derived from the file's own location, and `EMOTION_WHEEL_ROOT` / `DATA_DIR` already point inside this repository, so only these entries need repointing: * `PATH_TO_LLM`, `PATH_TO_VISUAL`, `PATH_TO_AUDIO` โ Qwen2.5-7B-Instruct, CLIP ViT-L/14 and chinese-hubert-large. * `BERT_BASE_UNCASED` โ the `bert-base-uncased` directory used by the Q-Former text encoder. * `PATH_TO_RAW_VIDEO`, `PATH_TO_RAW_AUDIO`, `PATH_TO_RAW_FACE`, `PATH_TO_TRANSCRIPTIONS` โ the media roots and the subtitle csv; all three datasets share one root. * `PATH_TO_LABEL` โ the per-dataset open-vocabulary label csvs, read from `dataset/` inside this repository. |
| The label files ship with the repository, so only the **raw media** has to be downloaded. Two options, as in [run-release.sh](run-release.sh): * **[Option 1]** download all of [MERChallenge/MER2025](https://huggingface.co/datasets/MERChallenge/MER2025). * **[Option 2]** download only `*/*_track2_train_human` and `*/*_track2_train_mercaptionplus` from [MERChallenge/MER2026](https://huggingface.co/datasets/MERChallenge/MER2026). Either way, keep everything under `dataset/`: the `audio/`, `video/` and `openface_face/` folders, all subtitles in `subtitle_chieng.csv`, and all the label files โ the three training parts `mercaptionplus-emotion.csv`, `mercaptionplus-description.csv` and `mer2025ov1000-emotion.csv`, plus the evaluation labels `ovmerdplus-emotion.csv` โ next to them. |
| Model Name | Model Type |
|---|---|
| clip-vit-large-patch14 | Visual Encoder |
| chinese-hubert-large | Audio Encoder |
| Qwen2.5-7B-Instruct | LLM |
| bert-base-uncased | Text Encoder / Tokenizer of the Q-Formers |
| mercaptionplus_outputhybird_bestsetup_bestfusion_face_r1stage1 | Stage 1 (cold start): trained on MER-Caption+ with pre-extracted face as input |
| mercaptionplus_outputhybird_bestsetup_bestfusion_face_r1stage2_mer25ov1000_continue60_g4_formatbeta00_ovlabel_numrestrict3 | Stage 2 (RL): GRPO optimization on MER2025-OV (P3 length penalty), continued from the Stage-1 epoch-60 checkpoint |
| We use MER-Caption+, a **large-scale** dataset, to equip the model with emotion understanding and format alignment. Because descriptive emotion datasets already follow a reasoning-like format, we treat the *description* as the thinking $o^t$ and the *open-vocabulary labels* as the answer $o^a$, unifying descriptive and open-vocabulary emotions into a single task. The released configuration uses `label_type: hybird`, i.e. each sample is rendered as a description, a bare OV label set, or a reasoning trace followed by an answer. |
| We use MER2025-OV, a **high-quality** dataset, for reward calculation and policy updates. Policy optimization follows **GRPO**: $G$ outputs are sampled for each prompt, and rewards are normalized within the group to produce relative advantages. The reward is the emotion-wheel metric itself, so this stage directly optimizes what is measured at test time. |
| **Variants.** Three length penalties $P_1$, $P_2$, $P_3$ were explored as separate architectures; this release ships only the loosest one, $P_3$, as [my_affectgpt/models/affectgpt_r1_noref_ovlabel4.py](my_affectgpt/models/affectgpt_r1_noref_ovlabel4.py), which is what the released Stage-2 config `..._numrestrict3.yaml` selects. |
| Inference results land in `output/results-{dataset}/{run_name}/`, one `.npz` file per epoch. The reward optimized during Stage 2 and the final metric are the **same** emotion-wheel (EW) measure โ both call `wheel_metric_calculation(..., level='level1')` in [my_affectgpt/evaluation/ew_metric.py](my_affectgpt/evaluation/ew_metric.py), which clusters prediction and ground truth onto five emotion wheels and averages the per-wheel F1. Scoring goes through three steps: 1. **The generated sentence is not a label set yet.** The model answers with a natural-language sentence (`The character's emotional state is ...`), so `main_zeroshot_scores` first calls Qwen2.5 (via vLLM) to extract the bare emotion words from each response, then maps every word to the emotion wheel along `word โ base form โ synonym โ wheel node` (`wheel_mapping.npz`). Words that cannot be resolved are dropped. 2. **Set-level comparison.** Prediction and ground truth are compared as *sets* of open-vocabulary labels โ duplicates removed โ and the overlap yields precision, recall and F1. 3. **Average over wheels.** The F1 scores are averaged across the five emotion wheels, giving $\text{EW}(\cdot,\cdot)$ โ the exact `accuracy_reward` maximized during RL. |
| This project is released under the Apache 2.0 license as found in the LICENSE file. The service is a research preview intended for **non-commercial use ONLY**. Please get in touch with us if you find any potential violations. |