Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy
On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding
Abstract (arXiv)
The development of affective multimodal language models (MLMs) has long been constrained by a gap between low-level perception and high-level interaction, leading to fragmented affective capabilities and limited generalization. To bridge this gap, we propose a cognitively inspired three-level hierarchy that organizes affective tasks according to their cognitive depth-perception, understanding, and interaction-and provides a unified conceptual foundation for advancing affective modeling. Guided by this hierarchy, we introduce Nano-EmoX, a small-scale multitask MLM, and P2E (Perception-to-Empathy), a curriculum-based training framework. Nano-EmoX integrates a suite of omni-modal encoders, including an enhanced facial encoder and a fusion encoder, to capture key multimodal affective cues and improve cross-task transferability. The outputs are projected into a unified language space via heterogeneous adapters, empowering a lightweight language model to tackle diverse affective tasks. Concurrently, P2E progressively cultivates emotional intelligence by aligning rapid perception with chain-of-thought-driven empathy. To the best of our knowledge, Nano-EmoX is the first compact MLM (2.2B) to unify six core affective tasks across all three hierarchy levels, achieving state-of-the-art or highly competitive performance across multiple benchmarks, demonstrating excellent efficiency and generalization. The code is available at https://github.com/waHAHJIAHAO/Nano-EmoX.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 77 references that are in the map. First posted on arXiv 2 Mar 2026.
Builds on
56 of 77 references.
Closest ancestors · 3
Same direction, newest first
- 2025R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning7 Mar
- 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models27 Jan
- 2024EmoLLM: Multimodal Emotional Understanding Meets Large Language Models24 Jun
Other references · 53
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023GPT-4 Technical Reportcited by 18,132
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
- 2025Qwen2.5-Omni Technical Reportcited by 704
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
38 more
- 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
- 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
- 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
- 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
- 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 117
- 2019Context-Aware Emotion Recognition Networkscited by 99
- 2021CEM: Commonsense-aware Empathetic Response Generationcited by 89
- 2022Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognitioncited by 81
- 2023Reasoning Implicit Sentiment with Chain-of-Thought Promptingcited by 79
- 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
- 2022FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videoscited by 69
- 2024EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysiscited by 62
- 2022Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planningcited by 59
- 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
- 2022M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Databasecited by 50
- 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
- 2022Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Modulecited by 43
- 2024EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuningcited by 42
- 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
- 2021Facial expression and attributes recognition based on multi-task learning of lightweight neural networkscited by 40
- 2019Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysiscited by 39
- 2024FaceXFormer: A Unified Transformer for Facial Analysiscited by 34
- 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
- 2023Learning Emotion Representations from Verbal and Nonverbal Communicationcited by 31
- 2024FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERscited by 30
- 2024MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversationscited by 25
- 2025Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysiscited by 25
- 2024MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognitioncited by 23
- 2024Towards Multimodal Emotional Support Conversation Systemscited by 22
- 2019Towards Understanding Neural Machine Translation with Word Importancecited by 19
- 2022Affection: Learning Affective Explanations for Real-World Visual Datacited by 19
- 2023Explainable Multimodal Emotion Recognitioncited by 18
- 2024OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognitioncited by 17
- 2025Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmarkcited by 17
- 2021Multi-Task Learning with Sentiment, Emotion, and Target Detection to Recognize Hate Speech and Offensive Languagecited by 12
- 2023CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognitioncited by 12
- 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 8
- 2024Unimodal Multi-Task Fusion for Emotional Mimicry Intensity Predictioncited by 6
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction
- 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
- 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models
- 2026EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding
- 2026AffectVerse: Emotional World Models for Multimodal Affective Computing
- 2025Empathic Prompting: Non-Verbal Context Integration for Multimodal LLM Conversations
- 2026E3mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment
- 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models