# Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities

arXiv: https://arxiv.org/abs/2605.16889 · HTML: https://mlsanity.com/paper/2605.16889/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 16 May 2026.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Multi-attention Recurrent Network for Human Communication Comprehension](https://mlsanity.com/paper/1802.00923/) (2018)
- [Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors](https://mlsanity.com/paper/1811.09362/) (2018)
- [Decoupled Multimodal Distilling for Emotion Recognition](https://mlsanity.com/paper/2303.13802/) (2023)
- [Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation](https://mlsanity.com/paper/2310.03986/) (2023)
- [Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2310.05804/) (2023)
- [Towards Robust Multimodal Sentiment Analysis with Incomplete Data](https://mlsanity.com/paper/2409.20012/) (2024)
- [DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis](https://mlsanity.com/paper/2412.12225/) (2024)
- [Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2407.05374/) (2024)
- [Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning](https://mlsanity.com/paper/2411.02793/) (2024)
- [Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models](https://mlsanity.com/paper/2602.07026/) (2026)
- [Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space](https://mlsanity.com/paper/2404.08923/) (2024)
- [Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis](https://mlsanity.com/paper/2211.13484/) (2022)
- [Anisotropic Modality Align](https://mlsanity.com/paper/2605.07825/) (2026)

## Built on by (most cited first, 1 of 1)

- [Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning](https://mlsanity.com/paper/2606.29334/) (2026)

## Abstract (arXiv)

Multimodal sentiment analysis relies on textual, acoustic, and visual signals, yet real-world data often suffer from modality missing and quality imbalance. Existing methods generate features for modality missing from available ones, but differences in expression mechanisms and sentiment dynamics across modalities may cause the generated features to deviate from true distributions and mislead prediction. In addition, unreliable modalities may dominate fusion, resulting in representation shift across modality combinations and unstable sentiment representations. To address these challenges, we propose a two-level reference alignment framework. The framework introduces stable references at the feature representation and sentiment decision levels to improve robustness under modality missing. First-level reference alignment leverages complete-modality samples to constrain representations and align different modality combinations into a shared sentiment space. Second-level reference alignment enforces cross-modal consistency at the decision level by suppressing unreliable modalities through prototype retrieval and voting. As a result, the framework maintains stable and reliable sentiment predictions under diverse missing-modality patterns. Experiments on CMU-MOSI and CMU-MOSEI show consistent improvements across various missing-modality settings. Under full-modality input, the proposed method achieves state-of-the-art performance, with ACC of 86.28% and 85.88%, and F1 of 86.24% and 85.86%.