# AffectGPT-R1: Leveraging Reinforcement Learning for Open-Vocabulary Multimodal Emotion Recognition

arXiv: https://arxiv.org/abs/2508.01318 · HTML: https://mlsanity.com/paper/2508.01318/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2025.

## Built on by (most cited first, 15 of 15)

- [MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models](https://mlsanity.com/paper/2508.09210/) (2025)
- [EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment](https://mlsanity.com/paper/2602.01173/) (2026)
- [AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization](https://mlsanity.com/paper/2602.07054/) (2026)
- [AffectVerse: Emotional World Models for Multimodal Affective Computing](https://mlsanity.com/paper/2605.19950/) (2026)
- [Omni-Perception Policy Optimization for Multimodal Emotion Reasoning](https://mlsanity.com/paper/2606.25325/) (2026)
- [Large Emotional World Model](https://mlsanity.com/paper/2512.24149/) (2025)
- [XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoning](https://mlsanity.com/paper/2602.05496/) (2026)
- [MER-R1: Multimodal Emotion Reasoning via Slow-Fast Thinking Synergy](https://mlsanity.com/paper/2606.27652/) (2026)
- [EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization](https://mlsanity.com/paper/2607.21013/) (2026)
- [ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoning](https://mlsanity.com/paper/2602.12714/) (2026)
- [C2F-Thinker: Coarse-to-Fine Reasoning with Hint-Guided Reinforcement Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2604.00013/) (2026)
- [Attribute-Grounded Selective Reasoning for Artwork Emotion Understanding with Multimodal Large Language Models](https://mlsanity.com/paper/2605.15755/) (2026)
- [Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow](https://mlsanity.com/paper/2605.21980/) (2026)
- [OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction](https://mlsanity.com/paper/2608.06013/) (2026)
- [Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Grounding](https://mlsanity.com/paper/2609.32804/) (2026)

## Abstract (arXiv)

Open-Vocabulary Multimodal Emotion Recognition (OV-MER) aims to predict emotions without being constrained by label spaces, enabling fine-grained emotion understanding. Unlike traditional discriminative methods, OV-MER leverages generative models to capture the full spectrum of emotions and employs emotion wheels (EWs) for metric calculation. Previous approaches (e.g., AffectGPT) primarily rely on token-level loss during training. However, this objective is misaligned with the metrics used in OV-MER, while these metrics cannot be optimized via gradient backpropagation. To address this limitation, we propose AffectGPT-R1, a reinforcement learning framework that treats EW-based metrics as a reward function and applies policy optimization to maximize this reward. Additionally, we introduce an explicit reasoning process and examine its necessity in OV-MER. To further guide model behavior, we incorporate auxiliary rewards that regularize both emotion reasoning and emotion prediction. We also apply length penalties to mitigate reward hacking. Experimental results demonstrate that AffectGPT-R1 yields significant performance improvements on OV-MER. Moreover, our approach enhances generalized emotion understanding, achieving state-of-the-art results on MER-UniBench. Our code is provided in the supplementary material and will be released to facilitate future research.