mlsanityData to 14 Oct 2026

SimMLM: A Simple Framework for Multi-modal Learning with Missing Modality

arXiv 2507.19264 ↗First posted 25 Jul 2025cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose SimMLM, a simple yet powerful framework for multimodal learning with missing modalities. Unlike existing approaches that rely on sophisticated network architectures or complex data imputation techniques, SimMLM provides a generic and effective solution that can adapt to various missing modality scenarios with improved accuracy and robustness. Specifically, SimMLM consists of a generic Dynamic Mixture of Modality Experts (DMoME) architecture, featuring a dynamic, learnable gating mechanism that automatically adjusts each modality's contribution in both full and partial modality settings. A key innovation of SimMLM is the proposed More vs. Fewer (MoFe) ranking loss, which ensures that task accuracy improves or remains stable as more modalities are made available. This aligns the model with an intuitive principle: removing one or more modalities should not increase accuracy. We validate SimMLM on multimodal medical image segmentation (BraTS 2018) and multimodal classification (UPMC Food-101, avMNIST) tasks, where it consistently surpasses competitive methods, demonstrating superior accuracy, interpretability, robustness, and reliability across both complete and missing modality scenarios at test time.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

31 of 47 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  9. 2019Measuring Calibration in Deep Learningcited by 446
  10. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
  11. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
  12. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 163
19 more
  1. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  2. 2022mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentationcited by 95
  3. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  4. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  5. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 70
  6. 2020Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusioncited by 68
  7. 2021Knowledge distillation from multi-modal to mono-modal segmentation networkscited by 68
  8. 2023M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalitiescited by 56
  9. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 46
  10. 2024Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Imagescited by 41
  11. 2023Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentationcited by 33
  12. 2024Modality-agnostic Domain Generalizable Medical Image Segmentation by Multi-Frequency in Multi-Scale Attentioncited by 26
  13. 2022ModDrop++: A Dynamic Filter Network with Intra-subject Co-training for Multiple Sclerosis Lesion Segmentation with Missing Modalitiescited by 22
  14. 2023Cross-modal Active Complementary Learning with Self-refining Correspondencecited by 22
  15. 2024Synthetic Data in AI: Challenges, Applications, and Ethical Implicationscited by 21
  16. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  17. 2023RankMixup: Ranking-Based Mixup Training for Network Calibrationcited by 19
  18. 2023Calibrating Multimodal Learningcited by 17
  19. 2024Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 20.

23 papers.

Sort:

By area:

  1. 2025Fusion or Confusion? Multimodal Complexity Is Not All You Needcited by 5
  2. 2026Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classificationcited by 4
  3. 2025Calibrated Multimodal Representation Learning with Missing Modalitiescited by 3
  4. 2025No Data? No Problem: Robust Vision-Tabular Learning with Missing Valuescited by 3
  5. 2026ModalImmune: Immunity Driven Unlearning via Self Destructive Trainingcited by 1
  6. 2026Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projectioncited by 1
  7. 2026Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modelingcited by 1
  8. 2026DepthAgent: Towards Better Universal Depth Estimation via Sample-wise Expert Selectioncited by 1
  9. 2026Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalitiescited by 1
  10. 2026ARMOR: A Robust Self-Supervised Framework for Root Cause Analysis in Microservices under Missing Modalitycited by 0
  11. 2026Maximizing T2-Only Prostate Cancer Localization from Expected Diffusion Weighted Imagingcited by 0
  12. 2026LARGO: Low-Rank Hypernetwork for Handling Missing Modalitiescited by 0
11 more
  1. 2026AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuningcited by 0
  2. 2026Unsupervised Learning for Missing Modalities in Multimodal Learningcited by 0
  3. 2026Unified Multimodal Model for Brain MRI Imputation and Understandingcited by 0
  4. 2026Spatio-Temporal Mixture-of-Modality-Experts Diffusion for Quantitative DCE-MRI Synthesis from Incomplete MR Sequencescited by 0
  5. 2026Residual-Guided Expert Specialization for Incomplete Multimodal Learningcited by 0
  6. 2026General Incomplete Multimodal Learning via Dynamic Quality Perceptioncited by 0
  7. 2026Co-Learning for Missing Arbitrary Modalities in Multi-modal Classificationcited by 0
  8. 2026Boundary-Aligned Contribution Routing for Robust Optical--SAR Object Detectioncited by 0
  9. 2026Whole-Body MRI Classification via Prompt-Based Clinical Conditioningcited by 0
  10. 2026PACER: Progressive Availability-Conditioned Evidence Routing for Radiology Report Generation under Incomplete Clinical Contextcited by 0
  11. 2026Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognitioncited by 0