mlsanityData to 14 Oct 2026

Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities

arXiv 2204.13707 ↗First posted 28 Apr 2022cs.LG cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

Multimodal sentiment analysis has been studied under the assumption that all modalities are available. However, such a strong assumption does not always hold in practice, and most of multimodal fusion models may fail when partial modalities are missing. Several works have addressed the missing modality problem; but most of them only considered the single modality missing case, and ignored the practically more general cases of multiple modalities missing. To this end, in this paper, we propose a Tag-Assisted Transformer Encoder (TATE) network to handle the problem of missing uncertain modalities. Specifically, we design a tag encoding module to cover both the single modality and multiple modalities missing cases, so as to guide the network's attention to those missing modalities. Besides, we adopt a new space projection pattern to align common vectors. Then, a Transformer encoder-decoder network is utilized to learn the missing modality features. At last, the outputs of the Transformer encoder are used for the final sentiment classification. Extensive experiments are conducted on CMU-MOSI and IEMOCAP datasets, showing that our method can achieve significant improvements compared with several baselines.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 24 of its 50 references that are in the map. First posted on arXiv 28 Apr 2022.

Builds on

24 of 50 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2022Generative Adversarial Networkscited by 5,627
  6. 2013Auto-Encoding Variational Bayescited by 4,053
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2020Reformer: The Efficient Transformercited by 1,668
  9. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  10. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  11. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 265
  12. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 180
12 more
  1. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  2. 2020Deep Partial Multi-View Learningcited by 71
  3. 2018Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learningcited by 43
  4. 2020TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysiscited by 40
  5. 2018Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modelingcited by 39
  6. 2020Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognitioncited by 36
  7. 2017VIGAN: Missing View Imputation with Generative Adversarial Networkscited by 33
  8. 2018Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Datacited by 29
  9. 2018Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselinescited by 23
  10. 2013Clustering on Multiple Incomplete Datasets via Collective Kernel Learningcited by 11
  11. 2021Iterative Network Pruning with Uncertainty Regularization for Lifelong Sentiment Classificationcited by 7
  12. 2021QuTI! Quantifying Text-Image Consistency in Multimodal Documentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 11; 2025: 7; 2026: 4.

28 papers.

Sort:

By area:

  1. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  2. 2023Single-Model and Any-Modality for Video Object Trackingcited by 65
  3. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
  4. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 18
  5. 2024Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learningcited by 16
  6. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 13
  7. 2023Learning Unseen Modality Interactioncited by 12
  8. 2023Improving Multimodal Sentiment Analysis: Supervised Angular Margin-based Contrastive Learning for Enhanced Fusion Representationcited by 12
  9. 2024MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modalitycited by 10
  10. 2025Knowledge Bridger: Towards Training-free Missing Modality Completioncited by 9
  11. 2024Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Datacited by 7
  12. 2024Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcitycited by 5
16 more
  1. 2023A Survey on Image-text Multimodal Modelscited by 4
  2. 2024Can Text-to-image Model Assist Multi-modal Learning for Visual Recognition with Visual Modality Missing?cited by 4
  3. 2024Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approachcited by 4
  4. 2025Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shiftscited by 4
  5. 2025RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognitioncited by 3
  6. 2025Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classificationcited by 2
  7. 2026Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learningcited by 2
  8. 2023Modality-Collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognitioncited by 1
  9. 2024AM^2-EmoJE: Adaptive Missing-Modality Emotion Recognition in Conversation via Joint Embedding Learningcited by 0
  10. 2024TI-ASU: Toward Robust Automatic Speech Understanding through Text-to-speech Imputation Against Missing Speech Modalitycited by 0
  11. 2025ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completioncited by 0
  12. 2025MoLAN: A Unified Modality-Aware Noise Dynamic Editing Framework for Multimodal Sentiment Analysiscited by 0
  13. 2025FSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysiscited by 0
  14. 2026OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflowscited by 0
  15. 2026Contrastive Mixed Prompt Learning for Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinationcited by 0
  16. 2026SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Datacited by 0