mlsanityData to 13 Oct 2026

Are Multimodal Transformers Robust to Missing Modality?

arXiv 2204.05454 ↗First posted 12 Apr 2022cs.CVcited by 184

Not on the map. Why?

Abstract (arXiv)

Multimodal data collected from the real world are often imperfect due to missing modalities. Therefore multimodal models that are robust against modal-incomplete data are highly preferred. Recently, Transformer models have shown great success in processing multimodal data. However, existing work has been limited to either architecture designs or pre-training strategies; whether Transformer models are naturally robust against missing-modal data has rarely been investigated. In this paper, we present the first-of-its-kind work to comprehensively investigate the behavior of Transformers in the presence of modal-incomplete data. Unsurprising, we find Transformer models are sensitive to missing modalities while different modal fusion strategies will significantly affect the robustness. What surprised us is that the optimal fusion strategy is dataset dependent even for the same Transformer model; there does not exist a universal strategy that works in general cases. Based on these findings, we propose a principle method to improve the robustness of Transformer models by automatically searching for an optimal fusion strategy regarding input data. Experimental validations on three benchmarks support the superior performance of the proposed method.

Key facts. Cited by 184 papers in the map (data through 13 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 12 Apr 2022.

Builds on

38 of 51 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  5. 2020Language Models are Few-Shot Learnerscited by 32,320
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,726
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,220
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,616
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,294
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,227
26 more
  1. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,142
  2. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  3. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  4. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 639
  5. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 486
  6. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 430
  7. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 428
  8. 2021Dynamic Neural Networks: A Surveycited by 406
  9. 2020Multi-modal Transformer for Video Retrievalcited by 395
  10. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  11. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  12. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 269
  13. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  14. 2017Gated Multimodal Units for Information Fusioncited by 246
  15. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 244
  16. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 187
  17. 2018Learning Factorized Multimodal Representationscited by 171
  18. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 166
  19. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  20. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  21. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  22. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  23. 2019MFAS: Multimodal Fusion Architecture Searchcited by 84
  24. 2020Parameter Efficient Multimodal Transformers for Video Representation Learningcited by 52
  25. 2020A Closer Look at the Robustness of Vision-and-Language Pre-trained Modelscited by 37
  26. 2016Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysiscited by 29

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 7; 2023: 33; 2024: 49; 2025: 44; 2026: 51.

184 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  3. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  4. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  5. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 91
  6. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 81
  7. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 75
  8. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 73
  9. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
  10. 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 53
  11. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 28
  12. 2025A Closer Look at Multimodal Representation Collapsecited by 28
38 more
  1. 2023MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognitioncited by 26
  2. 2023Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attentioncited by 25
  3. 2025SimMLM: A Simple Framework for Multi-modal Learning with Missing Modalitycited by 22
  4. 2024Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modalitycited by 21
  5. 2023Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognitioncited by 20
  6. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  7. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 18
  8. 2023Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrastcited by 17
  9. 2023Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarkscited by 17
  10. 2023Towards Robust Multimodal Prompting With Missing Modalitiescited by 16
  11. 2024Quantifying and Enhancing Multi-modal Robustness with Modality Preferencecited by 16
  12. 2024A Survey on Safe Multi-Modal Learning Systemcited by 15
  13. 2025Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learningcited by 15
  14. 2025What You Have is What You Track: Adaptive and Robust Multimodal Trackingcited by 15
  15. 2023Are Data-driven Explanations Robust against Out-of-distribution Data?cited by 14
  16. 2023MultiModN- Multimodal, Multi-Task, Interpretable Modular Networkscited by 14
  17. 2024Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modalitycited by 14
  18. 2024X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensingcited by 13
  19. 2024Promoting cross-modal representations to improve multimodal foundation models for physiological signalscited by 13
  20. 2025Robust Multimodal Survival Prediction with the Latent Differentiation Conditional Variational AutoEncodercited by 13
  21. 2023Learning Unseen Modality Interactioncited by 12
  22. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  23. 2024M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecturecited by 12
  24. 2024A Multimodal Single-Branch Embedding Network for Recommendation in Cold-Start and Missing Modality Scenarioscited by 12
  25. 2023LANISTR: Multimodal Learning from Structured and Unstructured Datacited by 11
  26. 2024Gradient-Guided Modality Decoupling for Missing-Modality Robustnesscited by 11
  27. 2023CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalitiescited by 10
  28. 2023Adventures of Trustworthy Vision-Language Models: A Surveycited by 10
  29. 2024MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modalitycited by 10
  30. 2024MMP: Towards Robust Multi-Modal Learning with Masked Modality Projectioncited by 10
  31. 2025MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Seriescited by 10
  32. 2023Rethinking Multimodal Content Moderation from an Asymmetric Angle with Mixed-modalitycited by 9
  33. 2025Knowledge Bridger: Towards Training-free Missing Modality Completioncited by 9
  34. 2025Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Datacited by 8
  35. 2024Exploring Missing Modality in Multimodal Egocentric Datasetscited by 7
  36. 2024LITE: Modeling Environmental Ecosystems with Multimodal Large Language Modelscited by 7
  37. 2024Beyond the Federation: Topology-aware Federated Learning for Generalization to Unseen Clientscited by 7
  38. 2023FRAMM: Fair Ranking with Missing Modalities for Clinical Trial Site Selectioncited by 6

50 most cited of 184 All 184 (JSON)