# Deep Multimodal Learning with Missing Modality: A Survey

arXiv: https://arxiv.org/abs/2409.07825 · HTML: https://mlsanity.com/paper/2409.07825/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- Builds on 119 of its 275 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Builds on (most cited first, 30 of 119)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Deep Learning for Person Re-identification: A Survey and Outlook](https://mlsanity.com/paper/2001.04193/) (2020)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Sketch-based Manga Retrieval using Manga109 Dataset](https://mlsanity.com/paper/1510.04389/) (2015)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://mlsanity.com/paper/2402.12226/) (2024)

## Built on by (most cited first, 30 of 92)

- [SimMLM: A Simple Framework for Multi-modal Learning with Missing Modality](https://mlsanity.com/paper/2507.19264/) (2025)
- [Towards Modality Generalization: A Benchmark and Prospective Analysis](https://mlsanity.com/paper/2412.18277/) (2024)
- [Multimodal Federated Learning With Missing Modalities through Feature Imputation Network](https://mlsanity.com/paper/2505.20232/) (2025)
- [Synergistic Prompting for Robust Visual Recognition with Missing Modalities](https://mlsanity.com/paper/2507.07802/) (2025)
- [Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention](https://mlsanity.com/paper/2603.27187/) (2026)
- [UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddings](https://mlsanity.com/paper/2505.11815/) (2025)
- [Cohort-Based Active Modality Acquisition](https://mlsanity.com/paper/2505.16791/) (2025)
- [Fusion or Confusion? Multimodal Complexity Is Not All You Need](https://mlsanity.com/paper/2512.22991/) (2025)
- [Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classification](https://mlsanity.com/paper/2601.22853/) (2026)
- [CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning](https://mlsanity.com/paper/2602.04920/) (2026)
- [Towards deployment-centric multimodal AI beyond vision and language](https://mlsanity.com/paper/2504.03603/) (2025)
- [Multimodal Representation Learning and Fusion](https://mlsanity.com/paper/2506.20494/) (2025)
- [Calibrated Multimodal Representation Learning with Missing Modalities](https://mlsanity.com/paper/2511.12034/) (2025)
- [ClusMFL: A Cluster-Enhanced Framework for Modality-Incomplete Multimodal Federated Learning in Brain Imaging Analysis](https://mlsanity.com/paper/2502.12180/) (2025)
- [Towards Robust Multimodal Representation: A Unified Approach with Adaptive Experts and Alignment](https://mlsanity.com/paper/2503.09498/) (2025)
- [Not Only Grey Matter: OmniBrain for Robust Multimodal Classification of Alzheimer's Disease](https://mlsanity.com/paper/2507.20872/) (2025)
- [Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration](https://mlsanity.com/paper/2510.19579/) (2025)
- [Multi-Modal Masked Autoencoders for Learning Image-Spectrum Associations for Galaxy Evolution and Cosmology](https://mlsanity.com/paper/2510.22527/) (2025)
- [A Multimodal Dataset of Student Oral Presentations with Sensors and Evaluation Data](https://mlsanity.com/paper/2601.07576/) (2026)
- [PyHealth 2.0: A Comprehensive Open-Source Toolkit for Accessible and Reproducible Clinical Deep Learning](https://mlsanity.com/paper/2601.16414/) (2026)
- [Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflection](https://mlsanity.com/paper/2602.16161/) (2026)
- [MetaPerch: Learning from metadata for bioacoustics foundation models](https://mlsanity.com/paper/2607.14072/) (2026)
- [Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities](https://mlsanity.com/paper/2504.08578/) (2025)
- [ICYM2I: The illusion of multimodal informativeness under missingness](https://mlsanity.com/paper/2505.16953/) (2025)
- [Multimodal Foundation Models for Early Disease Detection](https://mlsanity.com/paper/2510.01899/) (2025)
- [PhysioME: A Robust Multimodal Self-Supervised Framework for Physiological Signals with Missing Modalities](https://mlsanity.com/paper/2510.11110/) (2025)
- [OmniField: Conditioned Neural Fields for Robust Multimodal Spatiotemporal Learning](https://mlsanity.com/paper/2511.02205/) (2025)
- [ModalImmune: Immunity Driven Unlearning via Self Destructive Training](https://mlsanity.com/paper/2602.16197/) (2026)
- [REMIND: Rethinking Medical High-Modality Learning under Missingness--A Long-Tailed Distribution Perspective](https://mlsanity.com/paper/2603.00046/) (2026)
- [BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates](https://mlsanity.com/paper/2603.19718/) (2026)

## Abstract (arXiv)

During multimodal model training and testing, certain data modalities may be absent due to sensor limitations, cost constraints, privacy concerns, or data loss, negatively affecting performance. Multimodal learning techniques designed to handle missing modalities can mitigate this by ensuring model robustness even when some modalities are unavailable. This survey reviews recent progress in Multimodal Learning with Missing Modality (MLMM), focusing on deep learning methods. It provides the first comprehensive survey that covers the motivation and distinctions between MLMM and standard multimodal learning setups, followed by a detailed analysis of current methods, applications, and datasets, concluding with challenges and future directions.