# Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities

arXiv: https://arxiv.org/abs/2204.13707 · HTML: https://mlsanity.com/paper/2204.13707/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Deep Partial Multi-View Learning](https://mlsanity.com/paper/2011.06170/) (2020)
- [Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning](https://mlsanity.com/paper/1802.00924/) (2018)
- [TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2009.02902/) (2020)
- [Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling](https://mlsanity.com/paper/1806.06228/) (2018)
- [Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognition](https://mlsanity.com/paper/2010.00734/) (2020)
- [VIGAN: Missing View Imputation with Generative Adversarial Networks](https://mlsanity.com/paper/1708.06724/) (2017)
- [Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data](https://mlsanity.com/paper/1808.02096/) (2018)
- [Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselines](https://mlsanity.com/paper/1803.07427/) (2018)
- [Clustering on Multiple Incomplete Datasets via Collective Kernel Learning](https://mlsanity.com/paper/1310.1177/) (2013)
- [Iterative Network Pruning with Uncertainty Regularization for Lifelong Sentiment Classification](https://mlsanity.com/paper/2106.11197/) (2021)
- [QuTI! Quantifying Text-Image Consistency in Multimodal Documents](https://mlsanity.com/paper/2104.13748/) (2021)

## Built on by (most cited first, 28 of 28)

- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Single-Model and Any-Modality for Video Object Tracking](https://mlsanity.com/paper/2311.15851/) (2023)
- [Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities](https://mlsanity.com/paper/2404.16456/) (2024)
- [Deep Correlated Prompting for Visual Recognition with Missing Modalities](https://mlsanity.com/paper/2410.06558/) (2024)
- [Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning](https://mlsanity.com/paper/2411.02793/) (2024)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [Learning Unseen Modality Interaction](https://mlsanity.com/paper/2306.12795/) (2023)
- [Improving Multimodal Sentiment Analysis: Supervised Angular Margin-based Contrastive Learning for Enhanced Fusion Representation](https://mlsanity.com/paper/2312.02227/) (2023)
- [MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modality](https://mlsanity.com/paper/2409.04693/) (2024)
- [Knowledge Bridger: Towards Training-free Missing Modality Completion](https://mlsanity.com/paper/2502.19834/) (2025)
- [Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data](https://mlsanity.com/paper/2412.16243/) (2024)
- [Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcity](https://mlsanity.com/paper/2403.09428/) (2024)
- [A Survey on Image-text Multimodal Models](https://mlsanity.com/paper/2309.15857/) (2023)
- [Can Text-to-image Model Assist Multi-modal Learning for Visual Recognition with Visual Modality Missing?](https://mlsanity.com/paper/2402.09036/) (2024)
- [Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach](https://mlsanity.com/paper/2408.07445/) (2024)
- [Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts](https://mlsanity.com/paper/2506.10452/) (2025)
- [RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition](https://mlsanity.com/paper/2505.17501/) (2025)
- [Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classification](https://mlsanity.com/paper/2511.11460/) (2025)
- [Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning](https://mlsanity.com/paper/2602.01101/) (2026)
- [Modality-Collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition](https://mlsanity.com/paper/2312.15848/) (2023)
- [AM^2-EmoJE: Adaptive Missing-Modality Emotion Recognition in Conversation via Joint Embedding Learning](https://mlsanity.com/paper/2402.10921/) (2024)
- [TI-ASU: Toward Robust Automatic Speech Understanding through Text-to-speech Imputation Against Missing Speech Modality](https://mlsanity.com/paper/2404.17983/) (2024)
- [ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion](https://mlsanity.com/paper/2507.05624/) (2025)
- [MoLAN: A Unified Modality-Aware Noise Dynamic Editing Framework for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2508.09145/) (2025)
- [FSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysis](https://mlsanity.com/paper/2510.16086/) (2025)
- [OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows](https://mlsanity.com/paper/2602.04144/) (2026)
- [Contrastive Mixed Prompt Learning for Incomplete Multimodal Sentiment Analysis with Unseen Modality Combination](https://mlsanity.com/paper/2608.20019/) (2026)
- [SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data](https://mlsanity.com/paper/2609.30238/) (2026)

## Abstract (arXiv)

Multimodal sentiment analysis has been studied under the assumption that all modalities are available. However, such a strong assumption does not always hold in practice, and most of multimodal fusion models may fail when partial modalities are missing. Several works have addressed the missing modality problem; but most of them only considered the single modality missing case, and ignored the practically more general cases of multiple modalities missing. To this end, in this paper, we propose a Tag-Assisted Transformer Encoder (TATE) network to handle the problem of missing uncertain modalities. Specifically, we design a tag encoding module to cover both the single modality and multiple modalities missing cases, so as to guide the network's attention to those missing modalities. Besides, we adopt a new space projection pattern to align common vectors. Then, a Transformer encoder-decoder network is utilized to learn the missing modality features. At last, the outputs of the Transformer encoder are used for the final sentiment classification. Extensive experiments are conducted on CMU-MOSI and IEMOCAP datasets, showing that our method can achieve significant improvements compared with several baselines.