# Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy

arXiv: https://arxiv.org/abs/2603.02123 · HTML: https://mlsanity.com/paper/2603.02123/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 56 of its 77 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Multimodal emotion understanding](https://mlsanity.com/mllm/multimodal-emotion-understanding/)

## Builds on (most cited first, 30 of 56)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [MobileVLM V2: Faster and Stronger Baseline for Vision Language Model](https://mlsanity.com/paper/2402.03766/) (2024)
- [Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning](https://mlsanity.com/paper/2406.11161/) (2024)
- [R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning](https://mlsanity.com/paper/2503.05379/) (2025)
- [Context-Aware Emotion Recognition Networks](https://mlsanity.com/paper/1908.05913/) (2019)
- [CEM: Commonsense-aware Empathetic Response Generation](https://mlsanity.com/paper/2109.05739/) (2021)
- [Ola: Pushing the Frontiers of Omni-Modal Language Model](https://mlsanity.com/paper/2502.04328/) (2025)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [Reasoning Implicit Sentiment with Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.11255/) (2023)
- [FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2203.09463/) (2022)
- [EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysis](https://mlsanity.com/paper/2401.08508/) (2024)
- [Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planning](https://mlsanity.com/paper/2210.04242/) (2022)
- [EmoBench: Evaluating the Emotional Intelligence of Large Language Models](https://mlsanity.com/paper/2402.12071/) (2024)
- [M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database](https://mlsanity.com/paper/2205.10237/) (2022)
- [Multimodal Neurons in Pretrained Text-Only Transformers](https://mlsanity.com/paper/2308.01544/) (2023)
- [Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module](https://mlsanity.com/paper/2209.02604/) (2022)

## Built on by (most cited first, 1 of 1)

- [Affect-Prototype Guided Fusion for Open-Vocabulary Incomplete Multi-modal Emotion Recognition](https://mlsanity.com/paper/2609.16962/) (2026)

## Abstract (arXiv)

The development of affective multimodal language models (MLMs) has long been constrained by a gap between low-level perception and high-level interaction, leading to fragmented affective capabilities and limited generalization. To bridge this gap, we propose a cognitively inspired three-level hierarchy that organizes affective tasks according to their cognitive depth-perception, understanding, and interaction-and provides a unified conceptual foundation for advancing affective modeling. Guided by this hierarchy, we introduce Nano-EmoX, a small-scale multitask MLM, and P2E (Perception-to-Empathy), a curriculum-based training framework. Nano-EmoX integrates a suite of omni-modal encoders, including an enhanced facial encoder and a fusion encoder, to capture key multimodal affective cues and improve cross-task transferability. The outputs are projected into a unified language space via heterogeneous adapters, empowering a lightweight language model to tackle diverse affective tasks. Concurrently, P2E progressively cultivates emotional intelligence by aligning rapid perception with chain-of-thought-driven empathy. To the best of our knowledge, Nano-EmoX is the first compact MLM (2.2B) to unify six core affective tasks across all three hierarchy levels, achieving state-of-the-art or highly competitive performance across multiple benchmarks, demonstrating excellent efficiency and generalization. The code is available at https://github.com/waHAHJIAHAO/Nano-EmoX.