mlsanityData to 7 Oct 2026

Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognition

arXiv 2010.00734 ↗First posted 2 Oct 2020eess.AS cs.SD eess.IVcited by 36

Not on the map. Why?

Abstract (arXiv)

Automatic audio-visual expression recognition can play an important role in communication services such as tele-health, VOIP calls and human-machine interaction. Accuracy of audio-visual expression recognition could benefit from the interplay between the two modalities. However, most audio-visual expression recognition systems, trained in ideal conditions, fail to generalize in real world scenarios where either the audio or visual modality could be missing due to a number of reasons such as limited bandwidth, interactors' orientation, caller initiated muting. This paper studies the performance of a state-of-the art transformer when one of the modalities is missing. We conduct ablation studies to evaluate the model in the absence of either modality. Further, we propose a strategy to randomly ablate visual inputs during training at the clip or frame level to mimic real world scenarios. Results conducted on in-the-wild data, indicate significant generalization in proposed models trained on missing cues, with gains up to 17% for frame level ablations, showing that these training strategies cope better with the loss of input modalities.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 8; 2024: 13; 2025: 6; 2026: 3.

36 papers.

Sort:

By area:

  1. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  2. 2022GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversationcited by 86
  3. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  4. 2022Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalitiescited by 28
  5. 2023Multimodal Distillation for Egocentric Action Recognitioncited by 28
  6. 2022Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysiscited by 20
  7. 2023Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detectioncited by 13
  8. 2024Joint Multimodal Transformer for Emotion Recognition in the Wildcited by 13
  9. 2024SDR-GNN: Spectral Domain Reconstruction Graph Neural Network for Incomplete Multimodal Learning in Conversational Emotion Recognitioncited by 13
  10. 2023From the Lab to the Wild: Affect Modeling via Privileged Informationcited by 12
  11. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 12
  12. 2025Knowledge Bridger: Towards Training-free Missing Modality Completioncited by 9
24 more
  1. 2022M2R2: Missing-Modality Robust emotion Recognition framework with iterative data augmentationcited by 8
  2. 2024Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detectioncited by 8
  3. 2024Exploring Missing Modality in Multimodal Egocentric Datasetscited by 7
  4. 2023FRAMM: Fair Ranking with Missing Modalities for Clinical Trial Site Selectioncited by 6
  5. 2024MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoTcited by 6
  6. 2023Versatile audio-visual learning for emotion recognitioncited by 5
  7. 2025Foundation Models for CPS-IoT: Opportunities and Challengescited by 5
  8. 2024Knockout: A simple way to handle missing inputscited by 4
  9. 2024Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modalitycited by 4
  10. 2024Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approachcited by 4
  11. 2022A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognitioncited by 3
  12. 2023On Robustness to Missing Video for Audiovisual Speech Recognitioncited by 3
  13. 2023Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognitioncited by 2
  14. 2024IMAN: An Adaptive Network for Robust NPC Mortality Prediction with Missing Modalitiescited by 2
  15. 2025How Far Are We from Generating Missing Modalities with Foundation Models?cited by 1
  16. 2022Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modalitycited by 0
  17. 2024HyperMM : Robust Multimodal Learning with Varying-sized Inputscited by 0
  18. 2024Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalitiescited by 0
  19. 2025GSDNet: Revisiting Incomplete Multimodal-Diffusion from Graph Spectrum Perspective for Conversation Emotion Recognitioncited by 0
  20. 2025ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completioncited by 0
  21. 2025A Robust Incomplete Multimodal Low-Rank Adaptation Approach for Emotion Recognitioncited by 0
  22. 2026Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modelingcited by 0
  23. 2026Purify-then-Align: Towards Robust Human Sensing under Modality Missing with Knowledge Distillation from Noisy Multimodal Teachercited by 0
  24. 2026Resilient Vision-Tabular Multimodal Learning under Modality Missingnesscited by 0