Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
Not on the map. Why?
Abstract (arXiv)
Multimodal sentiment analysis (MSA) integrates heterogeneous text, audio, and visual signals to infer human emotions. While recent approaches leverage cross-modal complementarity, they often struggle to fully utilize weaker modalities. In practice, dominant modalities tend to overshadow non-verbal ones, inducing modality competition and limiting overall contributions. This imbalance degrades fusion performance and robustness under noisy or missing modalities. To address this, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC improves representation quality via semantic disentanglement and cross-modal enhancement, strengthening weaker modalities. To prevent dominant modalities from overwhelming others, an Energy-guided Modality Coordination mechanism achieves implicit gradient rebalancing via a differentiable equilibrium objective. Furthermore, Instance-aware Modality Trust Distillation estimates sample-level reliability to adaptively modulate fusion weights, ensuring robustness. Extensive experiments demonstrate that EBMC achieves state-of-the-art or competitive results and maintains strong performance under missing-modality settings.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 43 of its 74 references that are in the map. First posted on arXiv 14 Apr 2026.
Builds on
43 of 74 references.
References · 43
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
- 2022Multimodal Learning with Transformers: A Surveycited by 400
- 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
- 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 265
- 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
- 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
- 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 194
- 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 180
- 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
- 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 131
31 more
- 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
- 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 116
- 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 95
- 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
- 2022GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversationcited by 87
- 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
- 2024ESCoT: Towards Interpretable Emotional Support Dialogue Systemscited by 46
- 2022CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimationcited by 40
- 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
- 2023Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysiscited by 29
- 2024DPU: Dynamic Prototype Updating for Multimodal Out-of-Distribution Detectioncited by 29
- 2024Towards Robust Multimodal Sentiment Analysis with Incomplete Datacited by 27
- 2024DLF: Disentangled-Language-Focused Multimodal Sentiment Analysiscited by 27
- 2024Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Contentcited by 25
- 2025Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisitioncited by 19
- 2024AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignmentcited by 17
- 2024Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognitioncited by 15
- 2023Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognitioncited by 13
- 2024Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysiscited by 12
- 2024Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignmentcited by 12
- 2024Detached and Interactive Multimodal Learningcited by 11
- 2025Zero-Shot Conversational Stance Detection: Dataset and Approachescited by 11
- 2023Revisiting Modality Imbalance In Multimodal Pedestrian Detectioncited by 10
- 2025Hyperdimensional Uncertainty Quantification for Multimodal Uncertainty Fusion in Autonomous Vehicles Perceptioncited by 6
- 2024Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Promptingcited by 5
- 2024Semi-IIN: Semi-supervised Intra-inter modal Interaction Learning Network for Multimodal Sentiment Analysiscited by 5
- 2023ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysiscited by 4
- 2024Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attentioncited by 2
- 2025DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learningcited by 2
- 2025PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment Analysiscited by 2
- 2023Robust Multiview Multimodal Driver Monitoring System Using Masked Multi-Head Self-Attentioncited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.