Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoning
On the map: LLM reasoning → Audio-language model reasoning
Abstract (arXiv)
Despite rapid progress in multimodal large language models (MLLMs), their capability for deep emotional understanding remains limited. We argue that genuine affective intelligence requires explicit modeling of Theory of Mind (ToM), the cognitive substrate from which emotions arise. To this end, we introduce HitEmotion, a ToM-grounded hierarchical benchmark that diagnoses capability breakpoints across increasing levels of cognitive depth. Second, we propose a ToM-guided reasoning chain that tracks mental states and calibrates cross-modal evidence to achieve faithful emotional reasoning. We further introduce TMPO, a reinforcement learning method that uses intermediate mental states as process-level supervision to guide and strengthen model reasoning. Extensive experiments show that HitEmotion exposes deep emotional reasoning deficits in state-of-the-art models, especially on cognitively demanding tasks. In evaluation, the ToM-guided reasoning chain and TMPO improve end-task accuracy and yield more faithful, more coherent rationales. In conclusion, our work provides the research community with a practical toolkit for evaluating and enhancing the cognition-based emotional understanding capabilities of MLLMs. Our dataset and code are available at: https://HitEmotion.github.io/.
Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 98 references that are in the map. First posted on arXiv 1 Feb 2026.
Builds on
49 of 98 references.
Closest ancestors · 2
Same direction, newest first
- 2025Qwen2.5-Omni Technical Report26 Mar
- 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning17 Jun
Other references · 47
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024GPT-4o System Cardcited by 4,467
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
- 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
- 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
- 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
- 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
- 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
- 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
35 more
- 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
- 2025R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learningcited by 99
- 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
- 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
- 2018EmotionLines: An Emotion Corpus of Multi-Party Conversationscited by 80
- 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 75
- 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 70
- 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
- 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
- 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
- 2020Will-They-Won't-They: A Very Large Dataset for Stance Detection on Twittercited by 53
- 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
- 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
- 2021SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Imagescited by 49
- 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
- 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
- 2025AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Drivingcited by 32
- 2024EmoLLM: Multimodal Emotional Understanding Meets Large Language Modelscited by 31
- 2023SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Modelscited by 28
- 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 23
- 2024Multi-modal Stance Detection: New Datasets and Modelcited by 18
- 2024ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extractioncited by 17
- 2025Decompose-ToM: Enhancing Theory of Mind Reasoning in Large Language Models through Simulation and Task Decompositioncited by 17
- 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
- 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
- 2024Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Datasetcited by 15
- 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
- 2021ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptorscited by 12
- 2023Memotion 3: Dataset on Sentiment and Emotion Analysis of Codemixed Hindi-English Memescited by 12
- 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Taskscited by 11
- 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 10
- 2024MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Imagecited by 9
- 2025THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoningcited by 9
- 2024NUS-Emo at SemEval-2024 Task 3: Instruction-Tuning LLM for Multimodal Emotion-Cause Analysis in Conversationscited by 5
- 2025EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Modelscited by 5
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026CaC: Advancing Video Reward Models via Hierarchical Spatiotemporal Concentratingcited by 3
- 2026Story2Proposal: A Scaffold for Structured Scientific Paper Writingcited by 0
- 2026InsightVQA: High-Dimensional Emotion-Cognitive Visual Question Answering Benchmarkcited by 0
- 2026RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understandingcited by 0
- 2026From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Modelscited by 0
- 2026OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Modelscited by 0
- 2026Omni-IO Skills: Harnessing Your Agent Omni-Nativecited by 0
- 2026Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenescited by 0
- 2026Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenes8 Oct 2026
- 2026Omni-IO Skills: Harnessing Your Agent Omni-Native25 Sep 2026
- 2026OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models10 Sep 2026
- 2026From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models10 Sep 2026
- 2026RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding30 Aug 2026
- 2026InsightVQA: High-Dimensional Emotion-Cognitive Visual Question Answering Benchmark1 Jun 2026
- 2026CaC: Advancing Video Reward Models via Hierarchical Spatiotemporal Concentrating12 May 2026
- 2026Story2Proposal: A Scaffold for Structured Scientific Paper Writing28 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction
- 2026EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models
- 2026MER-R1: Multimodal Emotion Reasoning via Slow-Fast Thinking Synergy
- 2025Learning What to Attend First: Modality-Importance-Guided Reasoning for Reliable Multimodal Emotion Understanding
- 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner
- 2025Beyond Context to Cognitive Appraisal: Emotion Reasoning as a Theory of Mind Benchmark for Large Language Models
- 2024Through the Theory of Mind's Eye: Reading Minds with Multimodal Video Large Language Models
- 2025Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier