# SimMLM: A Simple Framework for Multi-modal Learning with Missing Modality

arXiv: https://arxiv.org/abs/2507.19264 · HTML: https://mlsanity.com/paper/2507.19264/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation](https://mlsanity.com/paper/2206.02425/) (2022)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [CentralNet: a Multilayer Approach for Multimodal Fusion](https://mlsanity.com/paper/1808.07275/) (2018)
- [Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts](https://mlsanity.com/paper/2410.08245/) (2024)
- [Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion](https://mlsanity.com/paper/2002.09708/) (2020)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)
- [M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2303.05302/) (2023)
- [MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models](https://mlsanity.com/paper/2407.12709/) (2024)
- [Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images](https://mlsanity.com/paper/2405.20469/) (2024)
- [Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation](https://mlsanity.com/paper/2306.02416/) (2023)
- [Modality-agnostic Domain Generalizable Medical Image Segmentation by Multi-Frequency in Multi-Scale Attention](https://mlsanity.com/paper/2405.06284/) (2024)
- [ModDrop++: A Dynamic Filter Network with Intra-subject Co-training for Multiple Sclerosis Lesion Segmentation with Missing Modalities](https://mlsanity.com/paper/2203.04959/) (2022)
- [Cross-modal Active Complementary Learning with Self-refining Correspondence](https://mlsanity.com/paper/2310.17468/) (2023)
- [Synthetic Data in AI: Challenges, Applications, and Ethical Implications](https://mlsanity.com/paper/2401.01629/) (2024)
- [Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models](https://mlsanity.com/paper/2407.12616/) (2024)
- [RankMixup: Ranking-Based Mixup Training for Network Calibration](https://mlsanity.com/paper/2308.11990/) (2023)
- [Calibrating Multimodal Learning](https://mlsanity.com/paper/2306.01265/) (2023)

## Built on by (most cited first, 23 of 23)

- [Fusion or Confusion? Multimodal Complexity Is Not All You Need](https://mlsanity.com/paper/2512.22991/) (2025)
- [Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classification](https://mlsanity.com/paper/2601.22853/) (2026)
- [Calibrated Multimodal Representation Learning with Missing Modalities](https://mlsanity.com/paper/2511.12034/) (2025)
- [No Data? No Problem: Robust Vision-Tabular Learning with Missing Values](https://mlsanity.com/paper/2512.19602/) (2025)
- [ModalImmune: Immunity Driven Unlearning via Self Destructive Training](https://mlsanity.com/paper/2602.16197/) (2026)
- [Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection](https://mlsanity.com/paper/2604.15856/) (2026)
- [Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modeling](https://mlsanity.com/paper/2604.18753/) (2026)
- [DepthAgent: Towards Better Universal Depth Estimation via Sample-wise Expert Selection](https://mlsanity.com/paper/2605.23281/) (2026)
- [Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalities](https://mlsanity.com/paper/2606.15514/) (2026)
- [ARMOR: A Robust Self-Supervised Framework for Root Cause Analysis in Microservices under Missing Modality](https://mlsanity.com/paper/2603.25538/) (2026)
- [Maximizing T2-Only Prostate Cancer Localization from Expected Diffusion Weighted Imaging](https://mlsanity.com/paper/2604.00985/) (2026)
- [LARGO: Low-Rank Hypernetwork for Handling Missing Modalities](https://mlsanity.com/paper/2605.06086/) (2026)
- [AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning](https://mlsanity.com/paper/2605.24816/) (2026)
- [Unsupervised Learning for Missing Modalities in Multimodal Learning](https://mlsanity.com/paper/2606.15743/) (2026)
- [Unified Multimodal Model for Brain MRI Imputation and Understanding](https://mlsanity.com/paper/2606.16484/) (2026)
- [Spatio-Temporal Mixture-of-Modality-Experts Diffusion for Quantitative DCE-MRI Synthesis from Incomplete MR Sequences](https://mlsanity.com/paper/2606.25535/) (2026)
- [Residual-Guided Expert Specialization for Incomplete Multimodal Learning](https://mlsanity.com/paper/2606.30355/) (2026)
- [General Incomplete Multimodal Learning via Dynamic Quality Perception](https://mlsanity.com/paper/2607.06943/) (2026)
- [Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification](https://mlsanity.com/paper/2607.24683/) (2026)
- [Boundary-Aligned Contribution Routing for Robust Optical--SAR Object Detection](https://mlsanity.com/paper/2608.15261/) (2026)
- [Whole-Body MRI Classification via Prompt-Based Clinical Conditioning](https://mlsanity.com/paper/2608.30824/) (2026)
- [PACER: Progressive Availability-Conditioned Evidence Routing for Radiology Report Generation under Incomplete Clinical Context](https://mlsanity.com/paper/2609.34487/) (2026)
- [Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition](https://mlsanity.com/paper/2610.15891/) (2026)

## Abstract (arXiv)

In this paper, we propose SimMLM, a simple yet powerful framework for multimodal learning with missing modalities. Unlike existing approaches that rely on sophisticated network architectures or complex data imputation techniques, SimMLM provides a generic and effective solution that can adapt to various missing modality scenarios with improved accuracy and robustness. Specifically, SimMLM consists of a generic Dynamic Mixture of Modality Experts (DMoME) architecture, featuring a dynamic, learnable gating mechanism that automatically adjusts each modality's contribution in both full and partial modality settings. A key innovation of SimMLM is the proposed More vs. Fewer (MoFe) ranking loss, which ensures that task accuracy improves or remains stable as more modalities are made available. This aligns the model with an intuitive principle: removing one or more modalities should not increase accuracy. We validate SimMLM on multimodal medical image segmentation (BraTS 2018) and multimodal classification (UPMC Food-101, avMNIST) tasks, where it consistently surpasses competitive methods, demonstrating superior accuracy, interpretability, robustness, and reliability across both complete and missing modality scenarios at test time.