mlsanityData to 7 Oct 2026

On Robustness in Multimodal Learning

arXiv 2304.04385 ↗First posted 10 Apr 2023cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Multimodal learning is defined as learning over multiple heterogeneous input modalities such as video, audio, and text. In this work, we are concerned with understanding how models behave as the type of modalities differ between training and deployment, a situation that naturally arises in many applications of multimodal learning to hardware platforms. We present a multimodal robustness framework to provide a systematic analysis of common multimodal representation learning methods. Further, we identify robustness short-comings of these approaches and propose two intervention techniques leading to 1.5×-4× robustness improvements on three datasets, AudioSet, Kinetics-400 and ImageNet-Captions. Finally, we demonstrate that these interventions better utilize additional modalities, if present, to achieve competitive results of 44.2 mAP on AudioSet 20K.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 52 of its 64 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

52 of 64 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2016Learning without Forgettingcited by 2,465
  12. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
38 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,894
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  5. 2016Deep Networks with Stochastic Depthcited by 973
  6. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  7. 2021Robust fine-tuning of zero-shot modelscited by 785
  8. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  9. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  11. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  12. 2018End-to-End Incremental Learningcited by 596
  13. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  14. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  15. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  16. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  17. 2018Generating Natural Language Adversarial Examplescited by 422
  18. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  19. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  20. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  21. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  22. 2022Masked Autoencoders that Listencited by 271
  23. 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 268
  24. 2018Generalisation in humans and deep neural networkscited by 237
  25. 2022Patching open-vocabulary models by interpolating weightscited by 187
  26. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  27. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  28. 2022(Certified!!) Adversarial Robustness for Free!cited by 130
  29. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  30. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  31. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  32. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  33. 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 61
  34. 2019Do Image Classifiers Generalize Across Time?cited by 59
  35. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 50
  36. 2022Audiovisual Masked Autoencoderscited by 47
  37. 2022Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-trainingcited by 32
  38. 2018Learn to Combine Modalities in Multimodal Deep Learningcited by 29

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 1.

3 papers.

  1. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 27
  2. 2025Multimodal Representation Learning and Fusioncited by 3
  3. 2024Multimodal CLIP Inference for Meta-Few-Shot Image Classificationcited by 0