# Unimodal-driven Distillation in Multimodal Emotion Recognition with Dynamic Fusion

arXiv: https://arxiv.org/abs/2503.23721 · HTML: https://mlsanity.com/paper/2503.23721/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)
- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/1908.11540/) (2019)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [MM-DFN: Multimodal Dynamic Fusion Network for Emotion Recognition in Conversations](https://mlsanity.com/paper/2203.02385/) (2022)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [Emotion Recognition in Speech using Cross-Modal Transfer in the Wild](https://mlsanity.com/paper/1808.05561/) (2018)
- [EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition](https://mlsanity.com/paper/2203.13504/) (2022)
- [HiGRU: Hierarchical Gated Recurrent Units for Utterance-level Emotion Recognition](https://mlsanity.com/paper/1904.04446/) (2019)
- [Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling](https://mlsanity.com/paper/1806.06228/) (2018)
- [A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.20494/) (2023)
- [Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2103.09154/) (2021)
- [Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition](https://mlsanity.com/paper/2201.05834/) (2022)
- [Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction](https://mlsanity.com/paper/2311.04507/) (2023)
- [Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillation](https://mlsanity.com/paper/2409.11018/) (2024)

## Built on by (most cited first, 4 of 4)

- [DistillDrive: End-to-End Multi-Mode Autonomous Driving Distillation by Isomorphic Hetero-Source Planning Model](https://mlsanity.com/paper/2508.05402/) (2025)
- [Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.13340/) (2026)
- [EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2607.18336/) (2026)
- [Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts](https://mlsanity.com/paper/2607.06611/) (2026)

## Abstract (arXiv)

Multimodal Emotion Recognition in Conversations (MERC) identifies emotional states across text, audio and video, which is essential for intelligent dialogue systems and opinion analysis. Existing methods emphasize heterogeneous modal fusion directly for cross-modal integration, but often suffer from disorientation in multimodal learning due to modal heterogeneity and lack of instructive guidance. In this work, we propose SUMMER, a novel heterogeneous multimodal integration framework leveraging Mixture of Experts with Hierarchical Cross-modal Fusion and Interactive Knowledge Distillation. Key components include a Sparse Dynamic Mixture of Experts (SDMoE) for capturing dynamic token-wise interactions, a Hierarchical Cross-Modal Fusion (HCMF) for effective fusion of heterogeneous modalities, and Interactive Knowledge Distillation (IKD), which uses a pre-trained unimodal teacher to guide multimodal fusion in latent and logit spaces. Experiments on IEMOCAP and MELD show SUMMER outperforms state-of-the-art methods, particularly in recognizing minority and semantically similar emotions.