# AffectGPT: Dataset and Framework for Explainable Multimodal Emotion Recognition

arXiv: https://arxiv.org/abs/2407.07653 · HTML: https://mlsanity.com/paper/2407.07653/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2024.

## Built on by (most cited first, 11 of 11)

- [Language Model Can Listen While Speaking](https://mlsanity.com/paper/2408.02622/) (2024)
- [HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context](https://mlsanity.com/paper/2506.21277/) (2025)
- [Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey](https://mlsanity.com/paper/2509.24322/) (2025)
- [MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models](https://mlsanity.com/paper/2508.09210/) (2025)
- [Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning](https://mlsanity.com/paper/2508.01181/) (2025)
- [DEEMO: De-identity Multimodal Emotion Recognition and Reasoning](https://mlsanity.com/paper/2504.19549/) (2025)
- [Rethinking Facial Expression Recognition in the Era of Multimodal Large Language Models: Benchmark, Datasets, and Beyond](https://mlsanity.com/paper/2511.00389/) (2025)
- [DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning](https://mlsanity.com/paper/2603.23916/) (2026)
- [MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models](https://mlsanity.com/paper/2605.14635/) (2026)
- [Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach](https://mlsanity.com/paper/2606.07585/) (2026)
- [GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?](https://mlsanity.com/paper/2610.15129/) (2026)

## Abstract (arXiv)

Explainable Multimodal Emotion Recognition (EMER) is an emerging task that aims to achieve reliable and accurate emotion recognition. However, due to the high annotation cost, the existing dataset (denoted as EMER-Fine) is small, making it difficult to perform supervised training. To reduce the annotation cost and expand the dataset size, this paper reviews the previous dataset construction process. Then, we simplify the annotation pipeline, avoid manual checks, and replace the closed-source models with open-source models. Finally, we build EMER-Coarse, a coarsely-labeled dataset containing large-scale samples. Besides the dataset, we propose a two-stage training framework AffectGPT. The first stage exploits EMER-Coarse to learn a coarse mapping between multimodal inputs and emotion-related descriptions; the second stage uses EMER-Fine to better align with manually-checked results. Experimental results demonstrate the effectiveness of our proposed method on the challenging EMER task. To facilitate further research, we will make the code and dataset available at: https://github.com/zeroQiaoba/AffectGPT.