# Decoupled Multimodal Distilling for Emotion Recognition

arXiv: https://arxiv.org/abs/2303.13802 · HTML: https://mlsanity.com/paper/2303.13802/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors](https://mlsanity.com/paper/1811.09362/) (2018)
- [Graph Distillation for Action Detection with Privileged Modalities](https://mlsanity.com/paper/1712.00108/) (2017)
- [Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification](https://mlsanity.com/paper/1804.10069/) (2018)
- [Knowledge Transfer Graph for Deep Collaborative Learning](https://mlsanity.com/paper/1909.04286/) (2019)

## Built on by (most cited first, 30 of 76)

- [Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning](https://mlsanity.com/paper/2406.11161/) (2024)
- [Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities](https://mlsanity.com/paper/2404.16456/) (2024)
- [Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model](https://mlsanity.com/paper/2405.18014/) (2024)
- [Towards Multimodal Sentiment Analysis Debiasing via Bias Purification](https://mlsanity.com/paper/2403.05023/) (2024)
- [Facial Affective Behavior Analysis with Instruction Tuning](https://mlsanity.com/paper/2404.05052/) (2024)
- [DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learning](https://mlsanity.com/paper/2503.11892/) (2025)
- [AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment](https://mlsanity.com/paper/2412.00833/) (2024)
- [Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning](https://mlsanity.com/paper/2411.02793/) (2024)
- [DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning](https://mlsanity.com/paper/2503.06456/) (2025)
- [Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2403.13659/) (2024)
- [Cohort-Individual Cooperative Learning for Multimodal Cancer Survival Analysis](https://mlsanity.com/paper/2404.02394/) (2024)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [Towards Robust Multimodal Physiological Foundation Models: Handling Arbitrary Missing Modalities](https://mlsanity.com/paper/2504.19596/) (2025)
- [Asynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations](https://mlsanity.com/paper/2407.04955/) (2024)
- [Multi-modal Knowledge Distillation-based Human Trajectory Forecasting](https://mlsanity.com/paper/2503.22201/) (2025)
- [MASA-TCN: Multi-anchor Space-aware Temporal Convolutional Neural Networks for Continuous and Discrete EEG Emotion Recognition](https://mlsanity.com/paper/2308.16207/) (2023)
- [Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysis](https://mlsanity.com/paper/2501.05474/) (2025)
- [Unbiased Dynamic Multimodal Fusion](https://mlsanity.com/paper/2603.19681/) (2026)
- [Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimation](https://mlsanity.com/paper/2403.11757/) (2024)
- [Tri-Subspaces Disentanglement for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2602.19585/) (2026)
- [TiCAL:Typicality-Based Consistency-Aware Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2511.15085/) (2025)
- [Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding](https://mlsanity.com/paper/2601.16449/) (2026)
- [CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning](https://mlsanity.com/paper/2602.19605/) (2026)
- [Triple Disentangled Representation Learning for Multimodal Affective Analysis](https://mlsanity.com/paper/2401.16119/) (2024)
- [Completed Feature Disentanglement Learning for Multimodal MRIs Analysis](https://mlsanity.com/paper/2407.04916/) (2024)
- [Resolving Sentiment Discrepancy for Multimodal Sentiment Detection via Semantics Completion and Decomposition](https://mlsanity.com/paper/2407.07026/) (2024)
- [Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts](https://mlsanity.com/paper/2506.10452/) (2025)
- [EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding](https://mlsanity.com/paper/2412.08049/) (2024)
- [Temporal-Spatial Decouple before Act: Disentangled Representation Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2601.13659/) (2026)
- [Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.13340/) (2026)

## Abstract (arXiv)

Human multimodal emotion recognition (MER) aims to perceive human emotions via language, visual and acoustic modalities. Despite the impressive performance of previous MER approaches, the inherent multimodal heterogeneities still haunt and the contribution of different modalities varies significantly. In this work, we mitigate this issue by proposing a decoupled multimodal distillation (DMD) approach that facilitates flexible and adaptive crossmodal knowledge distillation, aiming to enhance the discriminative features of each modality. Specially, the representation of each modality is decoupled into two parts, i.e., modality-irrelevant/-exclusive spaces, in a self-regression manner. DMD utilizes a graph distillation unit (GD-Unit) for each decoupled part so that each GD can be performed in a more specialized and effective manner. A GD-Unit consists of a dynamic graph where each vertice represents a modality and each edge indicates a dynamic knowledge distillation. Such GD paradigm provides a flexible knowledge transfer manner where the distillation weights can be automatically learned, thus enabling diverse crossmodal knowledge transfer patterns. Experimental results show DMD consistently obtains superior performance than state-of-the-art MER methods. Visualization results show the graph edges in DMD exhibit meaningful distributional patterns w.r.t. the modality-irrelevant/-exclusive feature spaces. Codes are released at urlhttps://github.com/mdswyz/DMD.