# Triple Disentangled Representation Learning for Multimodal Affective Analysis

arXiv: https://arxiv.org/abs/2401.16119 · HTML: https://mlsanity.com/paper/2401.16119/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Domain Separation Networks](https://mlsanity.com/paper/1608.06019/) (2016)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning](https://mlsanity.com/paper/1702.08811/) (2017)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis](https://mlsanity.com/paper/1911.05544/) (2019)
- [Decoupled Multimodal Distilling for Emotion Recognition](https://mlsanity.com/paper/2303.13802/) (2023)
- [Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research](https://mlsanity.com/paper/2005.00357/) (2020)
- [UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2211.11256/) (2022)
- [A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances](https://mlsanity.com/paper/2203.06935/) (2022)
- [M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues](https://mlsanity.com/paper/1911.05659/) (2019)
- [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.01797/) (2021)
- [Robust Learning with the Hilbert-Schmidt Independence Criterion](https://mlsanity.com/paper/1910.00270/) (2019)
- [Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition](https://mlsanity.com/paper/2201.05834/) (2022)
- [UAVM: Towards Unifying Audio and Visual Models](https://mlsanity.com/paper/2208.00061/) (2022)

## Built on by (most cited first, 4 of 4)

- [DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learning](https://mlsanity.com/paper/2503.11892/) (2025)
- [MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning](https://mlsanity.com/paper/2607.16789/) (2026)
- [Temporal Fusion Nexus: A task-agnostic multi-modal embedding model for clinical narratives and irregular time series in post-kidney transplant care](https://mlsanity.com/paper/2601.08503/) (2026)
- [IDRL: An Individual-Aware Multimodal Depression-Related Representation Learning Framework for Depression Diagnosis](https://mlsanity.com/paper/2603.11644/) (2026)

## Abstract (arXiv)

Multimodal learning has exhibited a significant advantage in affective analysis tasks owing to the comprehensive information of various modalities, particularly the complementary information. Thus, many emerging studies focus on disentangling the modality-invariant and modality-specific representations from input data and then fusing them for prediction. However, our study shows that modality-specific representations may contain information that is irrelevant or conflicting with the tasks, which downgrades the effectiveness of learned multimodal representations. We revisit the disentanglement issue, and propose a novel triple disentanglement approach, TriDiRA, which disentangles the modality-invariant, effective modality-specific and ineffective modality-specific representations from input data. By fusing only the modality-invariant and effective modality-specific representations, TriDiRA can significantly alleviate the impact of irrelevant and conflicting information across modalities during model training. Extensive experiments conducted on four benchmark datasets demonstrate the effectiveness and generalization of our triple disentanglement, which outperforms SOTA methods.