# OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition

arXiv: https://arxiv.org/abs/2410.01495 · HTML: https://mlsanity.com/paper/2410.01495/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2024.

## Built on by (most cited first, 17 of 17)

- [EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models](https://mlsanity.com/paper/2502.04424/) (2025)
- [EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models](https://mlsanity.com/paper/2505.11405/) (2025)
- [EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibration](https://mlsanity.com/paper/2512.15528/) (2025)
- [Omni-MMSI: Toward Identity-attributed Social Interaction Understanding](https://mlsanity.com/paper/2604.00267/) (2026)
- [Multimodal Video Emotion Recognition with Reliable Reasoning Priors](https://mlsanity.com/paper/2508.03722/) (2025)
- [AffectVerse: Emotional World Models for Multimodal Affective Computing](https://mlsanity.com/paper/2605.19950/) (2026)
- [Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy](https://mlsanity.com/paper/2603.02123/) (2026)
- [MOTOR-Bench: A Real-world Dataset and Multi-agent Framework for Zero-shot Human Mental State Understanding](https://mlsanity.com/paper/2605.09703/) (2026)
- [MODF-SIR: A Multi-agent Omni-modal Distilled Framework for Social Intelligence Reasoning](https://mlsanity.com/paper/2606.12018/) (2026)
- [EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization](https://mlsanity.com/paper/2607.21013/) (2026)
- [Contrastive Distillation of Emotion Knowledge from LLMs for Zero-Shot Emotion Recognition](https://mlsanity.com/paper/2505.18040/) (2025)
- [CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models](https://mlsanity.com/paper/2603.09993/) (2026)
- [Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.16463/) (2026)
- [MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models](https://mlsanity.com/paper/2605.14635/) (2026)
- [Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow](https://mlsanity.com/paper/2605.21980/) (2026)
- [COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning](https://mlsanity.com/paper/2609.22697/) (2026)
- [LongEmo: Towards Emotion Understanding and Reasoning in Long Videos](https://mlsanity.com/paper/2609.40079/) (2026)

## Abstract (arXiv)

Multimodal Emotion Recognition (MER) is a critical research area that seeks to decode human emotions from diverse data modalities. However, existing machine learning methods predominantly rely on predefined emotion taxonomies, which fail to capture the inherent complexity, subtlety, and multi-appraisal nature of human emotional experiences, as demonstrated by studies in psychology and cognitive science. To overcome this limitation, we advocate for introducing the concept of open vocabulary into MER. This paradigm shift aims to enable models to predict emotions beyond a fixed label space, accommodating a flexible set of categories to better reflect the nuanced spectrum of human emotions. To achieve this, we propose a novel paradigm: Open-Vocabulary MER (OV-MER), which enables emotion prediction without being confined to predefined spaces. However, constructing a dataset that encompasses the full range of emotions for OV-MER is practically infeasible; hence, we present a comprehensive solution including a newly curated database, novel evaluation metrics, and a preliminary benchmark. By advancing MER from basic emotions to more nuanced and diverse emotional states, we hope this work can inspire the next generation of MER, enhancing its generalizability and applicability in real-world scenarios. Code and dataset are available at: https://github.com/zeroQiaoba/AffectGPT.