# Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation

arXiv: https://arxiv.org/abs/2310.03986 · HTML: https://mlsanity.com/paper/2310.03986/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 96 references that are in the map.
- First posted on arXiv 6 Oct 2023.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning](https://mlsanity.com/paper/2210.08823/) (2022)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)

## Built on by (most cited first, 29 of 29)

- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities](https://mlsanity.com/paper/2407.11351/) (2024)
- [Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation](https://mlsanity.com/paper/2401.16923/) (2024)
- [MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection](https://mlsanity.com/paper/2410.03010/) (2024)
- [Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation](https://mlsanity.com/paper/2411.17141/) (2024)
- [Quantum Federated Learning for Multimodal Data: A Modality-Agnostic Approach](https://mlsanity.com/paper/2507.08217/) (2025)
- [MCE: Towards a General Framework for Handling Missing Modalities under Imbalanced Missing Rates](https://mlsanity.com/paper/2510.10534/) (2025)
- [Multimodal Representation Learning and Fusion](https://mlsanity.com/paper/2506.20494/) (2025)
- [Calibrated Multimodal Representation Learning with Missing Modalities](https://mlsanity.com/paper/2511.12034/) (2025)
- [Robust Multimodal Learning via Cross-Modal Proxy Tokens](https://mlsanity.com/paper/2501.17823/) (2025)
- [Toward Inherently Robust VLMs Against Visual Perception Attacks](https://mlsanity.com/paper/2506.11472/) (2025)
- [ModalImmune: Immunity Driven Unlearning via Self Destructive Training](https://mlsanity.com/paper/2602.16197/) (2026)
- [MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities](https://mlsanity.com/paper/2603.09874/) (2026)
- [Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection](https://mlsanity.com/paper/2604.15856/) (2026)
- [Towards Robust Federated Multimodal Graph Learning under Modality Heterogeneity](https://mlsanity.com/paper/2605.12584/) (2026)
- [Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities](https://mlsanity.com/paper/2605.16889/) (2026)
- [BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2506.03675/) (2025)
- [Missing-Modality-Aware Graph Neural Network for Cancer Classification](https://mlsanity.com/paper/2506.22901/) (2025)
- [RLBind: Adversarial-Invariant Cross-Modal Alignment for Unified Robust Embeddings](https://mlsanity.com/paper/2509.14383/) (2025)
- [Plug, Play, and Fortify: A Low-Cost Module for Robust Multimodal Image Understanding Models](https://mlsanity.com/paper/2602.22644/) (2026)
- [RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation](https://mlsanity.com/paper/2603.09149/) (2026)
- [Latent Space Guided Scenario Sampling for Multimodal Segmentation Under Missing Modalities](https://mlsanity.com/paper/2605.20372/) (2026)
- [PAMF: Prior-Aware Multimodal Fusion for Incomplete Time Series Data](https://mlsanity.com/paper/2606.06328/) (2026)
- [Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features](https://mlsanity.com/paper/2607.21347/) (2026)
- [Conformal Fusion Under Missing Modalities](https://mlsanity.com/paper/2608.07183/) (2026)
- [Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policies](https://mlsanity.com/paper/2609.03142/) (2026)
- [Residual Optimal Transport-Based Experts Collaboration Towards Modality-Aware Infrared-Visible Object Detection](https://mlsanity.com/paper/2609.03516/) (2026)
- [Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness](https://mlsanity.com/paper/2610.04792/) (2026)
- [Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition](https://mlsanity.com/paper/2610.15891/) (2026)

## Abstract (arXiv)

Multimodal learning seeks to utilize data from multiple sources to improve the overall performance of downstream tasks. It is desirable for redundancies in the data to make multimodal systems robust to missing or corrupted observations in some correlated modalities. However, we observe that the performance of several existing multimodal networks significantly deteriorates if one or multiple modalities are absent at test time. To enable robustness to missing modalities, we propose a simple and parameter-efficient adaptation procedure for pretrained multimodal networks. In particular, we exploit modulation of intermediate features to compensate for the missing modalities. We demonstrate that such adaptation can partially bridge performance drop due to missing modalities and outperform independent, dedicated networks trained for the available modality combinations in some cases. The proposed adaptation requires extremely small number of parameters (e.g., fewer than 1% of the total parameters) and applicable to a wide range of modality combinations and tasks. We conduct a series of experiments to highlight the missing modality robustness of our proposed method on five different multimodal tasks across seven datasets. Our proposed method demonstrates versatility across various tasks and datasets, and outperforms existing methods for robust multimodal learning with missing modalities.