mlsanityData to 7 Oct 2026

Can audio-visual integration strengthen robustness under multimodal attacks?

arXiv 2104.02000 ↗First posted 5 Apr 2021cs.CV cs.CR cs.SDcited by 23

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose to make a systematic study on machines multisensory perception under attacks. We use the audio-visual event recognition task against multimodal adversarial attacks as a proxy to investigate the robustness of audio-visual learning. We attack audio, visual, and both modalities to explore whether audio-visual integration still strengthens perception and how different fusion mechanisms affect the robustness of audio-visual models. For interpreting the multimodal interactions under attacks, we learn a weakly-supervised sound source visual localization model to localize sounding regions in videos. To mitigate multimodal attacks, we propose an audio-visual defense approach based on an audio-visual dissimilarity constraint and external feature memory banks. Extensive experiments demonstrate that audio-visual models are susceptible to multimodal adversarial attacks; audio-visual integration could decrease the model robustness rather than strengthen under multimodal attacks; even a weakly-supervised sound source visual localization model can be successfully fooled; our defense method can improve the invulnerability of audio-visual networks without significantly sacrificing clean model performance.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 62 of its 95 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

62 of 95 references.

References · 62

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  9. 2016Adversarial Machine Learning at Scalecited by 934
  10. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  11. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  12. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
38 more
  1. 2017Countering Adversarial Images using Input Transformationscited by 517
  2. 2017Look, Listen and Learncited by 452
  3. 2017Mitigating Adversarial Effects Through Randomizationcited by 382
  4. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  5. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  6. 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 268
  7. 2018The Sound of Pixelscited by 258
  8. 2017Objects that Soundcited by 254
  9. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  10. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  11. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  12. 2016Lip Reading Sentences in the Wildcited by 226
  13. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  14. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  15. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  16. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  17. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  18. 2018Lip Movements Generation at a Glancecited by 154
  19. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  20. 2019The Sound of Motionscited by 145
  21. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  22. 2019Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognitioncited by 123
  23. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  24. 2020Music Gesture for Visual Sound Separationcited by 112
  25. 2019Co-Separating Sounds of Visual Objectscited by 110
  26. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  27. 2017You said that?cited by 99
  28. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  29. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  30. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  31. 2016Ambient Sound Provides Supervision for Visual Learningcited by 77
  32. 2019Dual-modality seq2seq network for audio-visual event localizationcited by 72
  33. 2017DolphinAtack: Inaudible Voice Commandscited by 65
  34. 2017Deep Cross-Modal Audio-Visual Generationcited by 63
  35. 2015Distributional Smoothing with Virtual Adversarial Trainingcited by 61
  36. 2018Efficient Large-Scale Multi-Modal Classificationcited by 57
  37. 2019Adversarial Defense by Restricting the Hidden Space of Deep Neural Networkscited by 56
  38. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 6; 2024: 5; 2025: 4; 2026: 1.

23 papers.

Sort:

By area:

  1. 2022Sound Adversarial Audio-Visual Navigationcited by 46
  2. 2023Audio-Visual Class-Incremental Learningcited by 33
  3. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 30
  4. 2022Robustness Analysis of Video-Language Models Against Visual and Language Perturbationscited by 23
  5. 2022Visual Sound Localization in the Wild by Cross-Modal Interference Erasingcited by 19
  6. 2022Pay Self-Attention to Audio-Visual Navigationcited by 17
  7. 2024Quantifying and Enhancing Multi-modal Robustness with Modality Preferencecited by 16
  8. 2024A Survey on Safe Multi-Modal Learning Systemcited by 15
  9. 2024Bag of Tricks to Boost Adversarial Transferabilitycited by 12
  10. 2021Space-Time Memory Network for Sounding Object Localization in Videoscited by 10
  11. 2022Noise-Tolerant Learning for Audio-Visual Action Recognitioncited by 7
  12. 2021Understanding and Measuring Robustness of Multimodal Learningcited by 5
11 more
  1. 2023What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?cited by 5
  2. 2025Universal Adversarial Attack on Aligned Multimodal LLMscited by 5
  3. 2024MMCert: Provable Defense against Adversarial Attacks to Multi-modal Modelscited by 4
  4. 2023Instance-Level Trojan Attacks on Visual Question Answering via Adversarial Learning in Neuron Activation Spacecited by 3
  5. 2025Rethinking Audio-Visual Adversarial Vulnerability from Temporal and Modality Perspectivescited by 2
  6. 2025textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Timecited by 2
  7. 2023Looking and Listening: Audio Guided Text Recognitioncited by 1
  8. 2023Magmaw: Modality-Agnostic Adversarial Attacks on Machine Learning-Based Wireless Communication Systemscited by 1
  9. 2026SoundBreak: A Systematic Study of Audio-Only Adversarial Attacks on Trimodal Modelscited by 1
  10. 2024Adversarial Robustness in RGB-Skeleton Action Recognition: Leveraging Attention Modality Reweightercited by 0
  11. 2025TAIL: Text-Audio Incremental Learningcited by 0