mlsanityData to 7 Oct 2026

SMIL: Multimodal Learning with Severely Missing Modality

arXiv 2103.05677 ↗First posted 9 Mar 2021cs.CVcited by 243

Not on the map. Why?

Abstract (arXiv)

A common assumption in multimodal learning is the completeness of training data, i.e., full modalities are available in all training examples. Although there exists research endeavor in developing novel methods to tackle the incompleteness of testing data, e.g., modalities are partially missing in testing examples, few of them can handle incomplete training modalities. The problem becomes even more challenging if considering the case of severely missing, e.g., 90% training examples may have incomplete modalities. For the first time in the literature, this paper formally studies multimodal learning with missing modality in terms of flexibility (missing modalities in training, testing, or both) and efficiency (most training data have incomplete modality). Technically, we propose a new method named SMIL that leverages Bayesian meta-learning in uniformly achieving both objectives. To validate our idea, we conduct a series of experiments on three popular benchmarks: MM-IMDb, CMU-MOSI, and avMNIST. The results prove the state-of-the-art performance of SMIL over existing methods and generative baselines including autoencoders and generative adversarial networks. Our code is available at https://github.com/mengmenm/SMIL.

Key facts. Cited by 243 papers in the map (data through 7 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 9 Mar 2021.

Builds on

26 of 53 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  4. 2016Matching Networks for One Shot Learningcited by 2,246
  5. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  6. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  7. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  8. 2019Contrastive Representation Distillationcited by 734
  9. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  10. 2020Learning to Learn Single Domain Generalizationcited by 295
  11. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  12. 2018Probabilistic Model-Agnostic Meta-Learningcited by 254
14 more
  1. 2017Gated Multimodal Units for Information Fusioncited by 244
  2. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  3. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  4. 2018Learning Factorized Multimodal Representationscited by 171
  5. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  6. 2019Feature-Critic Networks for Heterogeneous Domain Generalizationcited by 130
  7. 2018Meta-Learning Probabilistic Inference For Predictioncited by 108
  8. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  9. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  10. 2019Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Taskscited by 48
  11. 2019ShapeMask: Learning to Segment Novel Objects by Refining Shape Priorscited by 42
  12. 2020Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledgecited by 35
  13. 2016Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysiscited by 29
  14. 2020Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Modelscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 17; 2023: 43; 2024: 53; 2025: 53; 2026: 68.

243 papers.

Sort:

By area:

  1. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  2. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 159
  3. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  4. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  5. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  6. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  7. 2022GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversationcited by 86
  8. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  9. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 74
  10. 2022M3Care: Learning with Missing Modalities in Multimodal Healthcare Datacited by 71
  11. 2023Single-Model and Any-Modality for Video Object Trackingcited by 64
  12. 2022Multimodal learning with graphscited by 54
38 more
  1. 2022MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray imagescited by 53
  2. 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 53
  3. 2024DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistencycited by 51
  4. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  5. 2021Uncertainty-guided Model Generalization to Unseen Domainscited by 41
  6. 2023Multimodal Variational Auto-encoder based Audio-Visual Segmentationcited by 37
  7. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  8. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
  9. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 27
  10. 2023MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognitioncited by 26
  11. 2024Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognitioncited by 26
  12. 2023Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attentioncited by 25
  13. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  14. 2025SimMLM: A Simple Framework for Multi-modal Learning with Missing Modalitycited by 22
  15. 2023Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignmentcited by 21
  16. 2024Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modalitycited by 21
  17. 2022Analyzing Modality Robustness in Multimodal Sentiment Analysiscited by 20
  18. 2023Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Reviewcited by 20
  19. 2023Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognitioncited by 20
  20. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  21. 2024IC3M: In-Car Multimodal Multi-object Monitoring for Abnormal Status of Both Driver and Passengerscited by 20
  22. 2025Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoningcited by 20
  23. 2022What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memescited by 18
  24. 2024All in One Framework for Multimodal Re-identification in the Wildcited by 18
  25. 2023Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognitioncited by 17
  26. 2023Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrastcited by 17
  27. 2023Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarkscited by 17
  28. 2024PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Ratescited by 17
  29. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 17
  30. 2024Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learningcited by 16
  31. 2025Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learningcited by 15
  32. 2025What You Have is What You Track: Adaptive and Robust Multimodal Trackingcited by 15
  33. 2023Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentationcited by 14
  34. 2023Are Data-driven Explanations Robust against Out-of-distribution Data?cited by 14
  35. 2023RedCore: Relative Advantage Aware Cross-modal Representation Learning for Missing Modalities with Imbalanced Missing Ratescited by 14
  36. 2024Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modalitycited by 14
  37. 2024DEAL: Disentangle and Localize Concept-level Explanations for VLMscited by 14
  38. 2024Graph Neural Networks in Histopathology: Emerging Trends and Future Directionscited by 13

50 most cited of 243 All 243 (JSON)