# Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognition

arXiv: https://arxiv.org/abs/2010.00734 · HTML: https://mlsanity.com/paper/2010.00734/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 30 of 36)

- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022)
- [FedMultimodal: A Benchmark For Multimodal Federated Learning](https://mlsanity.com/paper/2306.09486/) (2023)
- [Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities](https://mlsanity.com/paper/2204.13707/) (2022)
- [Multimodal Distillation for Egocentric Action Recognition](https://mlsanity.com/paper/2307.07483/) (2023)
- [Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2208.07589/) (2022)
- [Joint Multimodal Transformer for Emotion Recognition in the Wild](https://mlsanity.com/paper/2403.10488/) (2024)
- [Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection](https://mlsanity.com/paper/2306.02137/) (2023)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [SDR-GNN: Spectral Domain Reconstruction Graph Neural Network for Incomplete Multimodal Learning in Conversational Emotion Recognition](https://mlsanity.com/paper/2411.19822/) (2024)
- [From the Lab to the Wild: Affect Modeling via Privileged Information](https://mlsanity.com/paper/2305.10919/) (2023)
- [Knowledge Bridger: Towards Training-free Missing Modality Completion](https://mlsanity.com/paper/2502.19834/) (2025)
- [M2R2: Missing-Modality Robust emotion Recognition framework with iterative data augmentation](https://mlsanity.com/paper/2205.02524/) (2022)
- [Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection](https://mlsanity.com/paper/2402.05294/) (2024)
- [Exploring Missing Modality in Multimodal Egocentric Datasets](https://mlsanity.com/paper/2401.11470/) (2024)
- [FRAMM: Fair Ranking with Missing Modalities for Clinical Trial Site Selection](https://mlsanity.com/paper/2305.19407/) (2023)
- [MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT](https://mlsanity.com/paper/2411.12126/) (2024)
- [Versatile audio-visual learning for emotion recognition](https://mlsanity.com/paper/2305.07216/) (2023)
- [Foundation Models for CPS-IoT: Opportunities and Challenges](https://mlsanity.com/paper/2501.16368/) (2025)
- [Knockout: A simple way to handle missing inputs](https://mlsanity.com/paper/2405.20448/) (2024)
- [Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality](https://mlsanity.com/paper/2407.16171/) (2024)
- [Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach](https://mlsanity.com/paper/2408.07445/) (2024)
- [A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognition](https://mlsanity.com/paper/2210.10972/) (2022)
- [On Robustness to Missing Video for Audiovisual Speech Recognition](https://mlsanity.com/paper/2312.10088/) (2023)
- [Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition](https://mlsanity.com/paper/2311.10119/) (2023)
- [IMAN: An Adaptive Network for Robust NPC Mortality Prediction with Missing Modalities](https://mlsanity.com/paper/2410.18551/) (2024)
- [How Far Are We from Generating Missing Modalities with Foundation Models?](https://mlsanity.com/paper/2506.03530/) (2025)
- [Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality](https://mlsanity.com/paper/2210.03428/) (2022)
- [HyperMM : Robust Multimodal Learning with Varying-sized Inputs](https://mlsanity.com/paper/2407.20768/) (2024)
- [Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalities](https://mlsanity.com/paper/2408.14585/) (2024)

## Abstract (arXiv)

Automatic audio-visual expression recognition can play an important role in communication services such as tele-health, VOIP calls and human-machine interaction. Accuracy of audio-visual expression recognition could benefit from the interplay between the two modalities. However, most audio-visual expression recognition systems, trained in ideal conditions, fail to generalize in real world scenarios where either the audio or visual modality could be missing due to a number of reasons such as limited bandwidth, interactors' orientation, caller initiated muting. This paper studies the performance of a state-of-the art transformer when one of the modalities is missing. We conduct ablation studies to evaluate the model in the absence of either modality. Further, we propose a strategy to randomly ablate visual inputs during training at the clip or frame level to mimic real world scenarios. Results conducted on in-the-wild data, indicate significant generalization in proposed models trained on missing cues, with gains up to 17% for frame level ablations, showing that these training strategies cope better with the loss of input modalities.