mlsanityData to 7 Oct 2026

Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy

arXiv 2603.02123 ↗First posted 2 Mar 2026cs.AI cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

The development of affective multimodal language models (MLMs) has long been constrained by a gap between low-level perception and high-level interaction, leading to fragmented affective capabilities and limited generalization. To bridge this gap, we propose a cognitively inspired three-level hierarchy that organizes affective tasks according to their cognitive depth-perception, understanding, and interaction-and provides a unified conceptual foundation for advancing affective modeling. Guided by this hierarchy, we introduce Nano-EmoX, a small-scale multitask MLM, and P2E (Perception-to-Empathy), a curriculum-based training framework. Nano-EmoX integrates a suite of omni-modal encoders, including an enhanced facial encoder and a fusion encoder, to capture key multimodal affective cues and improve cross-task transferability. The outputs are projected into a unified language space via heterogeneous adapters, empowering a lightweight language model to tackle diverse affective tasks. Concurrently, P2E progressively cultivates emotional intelligence by aligning rapid perception with chain-of-thought-driven empathy. To the best of our knowledge, Nano-EmoX is the first compact MLM (2.2B) to unify six core affective tasks across all three hierarchy levels, achieving state-of-the-art or highly competitive performance across multiple benchmarks, demonstrating excellent efficiency and generalization. The code is available at https://github.com/waHAHJIAHAO/Nano-EmoX.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 77 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

56 of 77 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning7 Mar
  2. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models27 Jan
  3. 2024EmoLLM: Multimodal Emotional Understanding Meets Large Language Models24 Jun

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  11. 2025Qwen2.5-Omni Technical Reportcited by 704
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
38 more
  1. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  2. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  3. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  4. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  5. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 117
  6. 2019Context-Aware Emotion Recognition Networkscited by 99
  7. 2021CEM: Commonsense-aware Empathetic Response Generationcited by 89
  8. 2022Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognitioncited by 81
  9. 2023Reasoning Implicit Sentiment with Chain-of-Thought Promptingcited by 79
  10. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  11. 2022FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videoscited by 69
  12. 2024EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysiscited by 62
  13. 2022Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planningcited by 59
  14. 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
  15. 2022M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Databasecited by 50
  16. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  17. 2022Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Modulecited by 43
  18. 2024EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuningcited by 42
  19. 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
  20. 2021Facial expression and attributes recognition based on multi-task learning of lightweight neural networkscited by 40
  21. 2019Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysiscited by 39
  22. 2024FaceXFormer: A Unified Transformer for Facial Analysiscited by 34
  23. 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
  24. 2023Learning Emotion Representations from Verbal and Nonverbal Communicationcited by 31
  25. 2024FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERscited by 30
  26. 2024MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversationscited by 25
  27. 2025Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysiscited by 25
  28. 2024MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognitioncited by 23
  29. 2024Towards Multimodal Emotional Support Conversation Systemscited by 22
  30. 2019Towards Understanding Neural Machine Translation with Word Importancecited by 19
  31. 2022Affection: Learning Affective Explanations for Real-World Visual Datacited by 19
  32. 2023Explainable Multimodal Emotion Recognitioncited by 18
  33. 2024OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognitioncited by 17
  34. 2025Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmarkcited by 17
  35. 2021Multi-Task Learning with Sentiment, Emotion, and Target Detection to Recognize Hate Speech and Offensive Languagecited by 12
  36. 2023CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognitioncited by 12
  37. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 8
  38. 2024Unimodal Multi-Task Fusion for Emotional Mimicry Intensity Predictioncited by 6

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction
  2. 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
  3. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models
  4. 2026EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding
  5. 2026AffectVerse: Emotional World Models for Multimodal Affective Computing
  6. 2025Empathic Prompting: Non-Verbal Context Integration for Multimodal LLM Conversations
  7. 2026E3mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment
  8. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models