# MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection

arXiv: https://arxiv.org/abs/2410.03010 · HTML: https://mlsanity.com/paper/2410.03010/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 52 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 30 of 39)

- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation](https://mlsanity.com/paper/2308.02097/) (2023)
- [Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2007.09183/) (2020)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Delivering Arbitrary-Modal Semantic Segmentation](https://mlsanity.com/paper/2303.01480/) (2023)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation](https://mlsanity.com/paper/1907.11150/) (2019)
- [Towards Good Practices for Missing Modality Robust Action Recognition](https://mlsanity.com/paper/2211.13916/) (2022)
- [Unified Multi-Modal Image Synthesis for Missing Modality Imputation](https://mlsanity.com/paper/2304.05340/) (2023)
- [MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning](https://mlsanity.com/paper/2304.08028/) (2023)
- [Modality Dropout for Improved Performance-driven Talking Faces](https://mlsanity.com/paper/2005.13616/) (2020)
- [EmbraceNet: A robust deep learning architecture for multimodal classification](https://mlsanity.com/paper/1904.09078/) (2019)
- [Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation](https://mlsanity.com/paper/2310.03986/) (2023)
- [Complementary Random Masking for RGB-Thermal Semantic Segmentation](https://mlsanity.com/paper/2303.17386/) (2023)

## Built on by (most cited first, 10 of 10)

- [LLM/Agent-as-Data-Analyst: A Survey](https://mlsanity.com/paper/2509.23988/) (2025)
- [Fusion or Confusion? Multimodal Complexity Is Not All You Need](https://mlsanity.com/paper/2512.22991/) (2025)
- [Robust Multimodal Learning via Cross-Modal Proxy Tokens](https://mlsanity.com/paper/2501.17823/) (2025)
- [CLARGA: Multimodal Graph Representation Learning over Arbitrary Sets of Modalities](https://mlsanity.com/paper/2512.11901/) (2025)
- [MSNN-LINet: Cross-Modal Learning via Continuous Linear Integration](https://mlsanity.com/paper/2606.31135/) (2026)
- [Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features](https://mlsanity.com/paper/2607.21347/) (2026)
- [Compass: Degradation-Simulated Reciprocal Learning with Lightweight Needle RWKV for Multimodal Crack Segmentation under Missing Modalities](https://mlsanity.com/paper/2608.03559/) (2026)
- [Sequential Modality Dropout for Robust Multi-Modal Sequential Recommendation](https://mlsanity.com/paper/2608.10240/) (2026)
- [TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs](https://mlsanity.com/paper/2608.18386/) (2026)
- [ForceDelta-VLA: Distilling Force-Conditioned ActionCorrections for Contact-Rich Manipulation](https://mlsanity.com/paper/2609.18242/) (2026)

## Abstract (arXiv)

Multimodal learning seeks to combine data from multiple input sources to enhance the performance of different downstream tasks. In real-world scenarios, performance can degrade substantially if some input modalities are missing. Existing methods that can handle missing modalities involve custom training or adaptation steps for each input modality combination. These approaches are either tied to specific modalities or become computationally expensive as the number of input modalities increases. In this paper, we propose Masked Modality Projection (MMP), a method designed to train a single model that is robust to any missing modality scenario. We achieve this by randomly masking a subset of modalities during training and learning to project available input modalities to estimate the tokens for the masked modalities. This approach enables the model to effectively learn to leverage the information from the available modalities to compensate for the missing ones, enhancing missing modality robustness. We conduct a series of experiments with various baseline models and datasets to assess the effectiveness of this strategy. Experiments demonstrate that our approach improves robustness to different missing modality scenarios, outperforming existing methods designed for missing modalities or specific modality combinations.