# Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition

arXiv: https://arxiv.org/abs/2407.05374 · HTML: https://mlsanity.com/paper/2407.05374/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 7 Jul 2024.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries](https://mlsanity.com/paper/2008.09036/) (2020)
- [Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors](https://mlsanity.com/paper/1811.09362/) (2018)
- [Multimodal Language Analysis with Recurrent Multistage Fusion](https://mlsanity.com/paper/1808.03920/) (2018)
- [Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data](https://mlsanity.com/paper/1808.02096/) (2018)
- [Multimodal and Multi-view Models for Emotion Recognition](https://mlsanity.com/paper/1906.10198/) (2019)

## Built on by (most cited first, 26 of 26)

- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Classifier-guided Gradient Modulation for Enhanced Multimodal Learning](https://mlsanity.com/paper/2411.01409/) (2024)
- [Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises](https://mlsanity.com/paper/2503.02616/) (2025)
- [Knowledge Bridger: Towards Training-free Missing Modality Completion](https://mlsanity.com/paper/2502.19834/) (2025)
- [ConceptGuard: Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation](https://mlsanity.com/paper/2503.10358/) (2025)
- [Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning](https://mlsanity.com/paper/2506.09473/) (2025)
- [Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts](https://mlsanity.com/paper/2506.10452/) (2025)
- [Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision](https://mlsanity.com/paper/2504.21423/) (2025)
- [Towards Robust and Realible Multimodal Misinformation Recognition with Incomplete Modality](https://mlsanity.com/paper/2510.05839/) (2025)
- [DCER: Dual-Stage Compression and Energy-Based Reconstruction](https://mlsanity.com/paper/2602.04904/) (2026)
- [How Far Are We from Generating Missing Modalities with Foundation Models?](https://mlsanity.com/paper/2506.03530/) (2025)
- [ModalImmune: Immunity Driven Unlearning via Self Destructive Training](https://mlsanity.com/paper/2602.16197/) (2026)
- [Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Data](https://mlsanity.com/paper/2603.02695/) (2026)
- [POLY-SIM: Polyglot Speaker Identification with Missing Modality Grand Challenge 2026 Evaluation Plan](https://mlsanity.com/paper/2603.24569/) (2026)
- [Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities](https://mlsanity.com/paper/2604.05558/) (2026)
- [Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities](https://mlsanity.com/paper/2605.16889/) (2026)
- [Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation](https://mlsanity.com/paper/2504.13465/) (2025)
- [Contextualized Multimodal Lifelong Person Re-Identification in Hybrid Clothing States](https://mlsanity.com/paper/2509.11247/) (2025)
- [Buffer replay enhances the robustness of multimodal learning under missing-modality](https://mlsanity.com/paper/2511.23070/) (2025)
- [OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows](https://mlsanity.com/paper/2602.04144/) (2026)
- [AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.10043/) (2026)
- [Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2605.21565/) (2026)
- [Unified Multimodal Model for Brain MRI Imputation and Understanding](https://mlsanity.com/paper/2606.16484/) (2026)
- [Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features](https://mlsanity.com/paper/2607.21347/) (2026)
- [Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy Correction](https://mlsanity.com/paper/2608.19971/) (2026)
- [Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients](https://mlsanity.com/paper/2608.21941/) (2026)

## Abstract (arXiv)

The development of multimodal models has significantly advanced multimodal sentiment analysis and emotion recognition. However, in real-world applications, the presence of various missing modality cases often leads to a degradation in the model's performance. In this work, we propose a novel multimodal Transformer framework using prompt learning to address the issue of missing modalities. Our method introduces three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts. These prompts enable the generation of missing modality features and facilitate the learning of intra- and inter-modality information. Through prompt learning, we achieve a substantial reduction in the number of trainable parameters. Our proposed method outperforms other methods significantly across all evaluation metrics. Extensive experiments and ablation studies are conducted to demonstrate the effectiveness and robustness of our method, showcasing its ability to effectively handle missing modalities.