mlsanityData to 7 Oct 2026

Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoning

arXiv 2602.00971 ↗First posted 1 Feb 2026cs.CVcited by 8

On the map: LLM reasoning → Audio-language model reasoning

also in 1 more
Abstract (arXiv)

Despite rapid progress in multimodal large language models (MLLMs), their capability for deep emotional understanding remains limited. We argue that genuine affective intelligence requires explicit modeling of Theory of Mind (ToM), the cognitive substrate from which emotions arise. To this end, we introduce HitEmotion, a ToM-grounded hierarchical benchmark that diagnoses capability breakpoints across increasing levels of cognitive depth. Second, we propose a ToM-guided reasoning chain that tracks mental states and calibrates cross-modal evidence to achieve faithful emotional reasoning. We further introduce TMPO, a reinforcement learning method that uses intermediate mental states as process-level supervision to guide and strengthen model reasoning. Extensive experiments show that HitEmotion exposes deep emotional reasoning deficits in state-of-the-art models, especially on cognitively demanding tasks. In evaluation, the ToM-guided reasoning chain and TMPO improve end-task accuracy and yield more faithful, more coherent rationales. In conclusion, our work provides the research community with a practical toolkit for evaluating and enhancing the cognition-based emotional understanding capabilities of MLLMs. Our dataset and code are available at: https://HitEmotion.github.io/.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 98 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

49 of 98 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Qwen2.5-Omni Technical Report26 Mar
  2. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning17 Jun

Other references · 47

  1. 2025Qwen2.5-VL Technical Reportcited by 5,418
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  8. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  9. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  10. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  11. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  12. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
35 more
  1. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
  2. 2025R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learningcited by 99
  3. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  4. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  5. 2018EmotionLines: An Emotion Corpus of Multi-Party Conversationscited by 80
  6. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 75
  7. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 70
  8. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
  9. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  10. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  11. 2020Will-They-Won't-They: A Very Large Dataset for Stance Detection on Twittercited by 53
  12. 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
  13. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
  14. 2021SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Imagescited by 49
  15. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  16. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  17. 2025AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Drivingcited by 32
  18. 2024EmoLLM: Multimodal Emotional Understanding Meets Large Language Modelscited by 31
  19. 2023SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Modelscited by 28
  20. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 23
  21. 2024Multi-modal Stance Detection: New Datasets and Modelcited by 18
  22. 2024ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extractioncited by 17
  23. 2025Decompose-ToM: Enhancing Theory of Mind Reasoning in Large Language Models through Simulation and Task Decompositioncited by 17
  24. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  25. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  26. 2024Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Datasetcited by 15
  27. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  28. 2021ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptorscited by 12
  29. 2023Memotion 3: Dataset on Sentiment and Emotion Analysis of Codemixed Hindi-English Memescited by 12
  30. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Taskscited by 11
  31. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 10
  32. 2024MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Imagecited by 9
  33. 2025THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoningcited by 9
  34. 2024NUS-Emo at SemEval-2024 Task 3: Instruction-Tuning LLM for Multimodal Emotion-Cause Analysis in Conversationscited by 5
  35. 2025EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction
  2. 2026EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models
  3. 2026MER-R1: Multimodal Emotion Reasoning via Slow-Fast Thinking Synergy
  4. 2025Learning What to Attend First: Modality-Importance-Guided Reasoning for Reliable Multimodal Emotion Understanding
  5. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner
  6. 2025Beyond Context to Cognitive Appraisal: Emotion Reasoning as a Theory of Mind Benchmark for Large Language Models
  7. 2024Through the Theory of Mind's Eye: Reading Minds with Multimodal Video Large Language Models
  8. 2025Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier