mlsanityData to 14 Oct 2026

Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition

arXiv 2407.05374 ↗First posted 7 Jul 2024cs.CL cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

The development of multimodal models has significantly advanced multimodal sentiment analysis and emotion recognition. However, in real-world applications, the presence of various missing modality cases often leads to a degradation in the model's performance. In this work, we propose a novel multimodal Transformer framework using prompt learning to address the issue of missing modalities. Our method introduces three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts. These prompts enable the generation of missing modality features and facilitate the learning of intra- and inter-modality information. Through prompt learning, we achieve a substantial reduction in the number of trainable parameters. Our proposed method outperforms other methods significantly across all evaluation metrics. Extensive experiments and ablation studies are conducted to demonstrate the effectiveness and robustness of our method, showcasing its ability to effectively handle missing modalities.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 7 Jul 2024.

Builds on

19 of 31 references.

References · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  8. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  9. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 265
  10. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  11. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
  12. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
7 more
  1. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 131
  2. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  3. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  4. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  5. 2018Multimodal Language Analysis with Recurrent Multistage Fusioncited by 44
  6. 2018Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Datacited by 29
  7. 2019Multimodal and Multi-view Models for Emotion Recognitioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 11; 2026: 13.

26 papers.

Sort:

By area:

  1. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  2. 2024Classifier-guided Gradient Modulation for Enhanced Multimodal Learningcited by 35
  3. 2025Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noisescited by 10
  4. 2025Knowledge Bridger: Towards Training-free Missing Modality Completioncited by 9
  5. 2025ConceptGuard: Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigationcited by 7
  6. 2025Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learningcited by 7
  7. 2025Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shiftscited by 4
  8. 2025Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervisioncited by 3
  9. 2025Towards Robust and Realible Multimodal Misinformation Recognition with Incomplete Modalitycited by 2
  10. 2026DCER: Dual-Stage Compression and Energy-Based Reconstructioncited by 2
  11. 2025How Far Are We from Generating Missing Modalities with Foundation Models?cited by 1
  12. 2026ModalImmune: Immunity Driven Unlearning via Self Destructive Trainingcited by 1
14 more
  1. 2026Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Datacited by 1
  2. 2026POLY-SIM: Polyglot Speaker Identification with Missing Modality Grand Challenge 2026 Evaluation Plancited by 1
  3. 2026Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalitiescited by 1
  4. 2026Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalitiescited by 1
  5. 2025Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimationcited by 0
  6. 2025Contextualized Multimodal Lifelong Person Re-Identification in Hybrid Clothing Statescited by 0
  7. 2025Buffer replay enhances the robustness of multimodal learning under missing-modalitycited by 0
  8. 2026OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflowscited by 0
  9. 2026AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognitioncited by 0
  10. 2026Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognitioncited by 0
  11. 2026Unified Multimodal Model for Brain MRI Imputation and Understandingcited by 0
  12. 2026Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Featurescited by 0
  13. 2026Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy Correctioncited by 0
  14. 2026Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patientscited by 0