mlsanityData to 7 Oct 2026

Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection

arXiv 2210.00753 ↗First posted 3 Oct 2022cs.SD cs.LG cs.MMcited by 3

Not on the map. Why?

Abstract (arXiv)

Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications. However, to the best of our knowledge, the adversarial robustness of AVASD models hasn't been investigated, not to mention the effective defense against such attacks. In this paper, we are the first to reveal the vulnerability of AVASD models under audio-only, visual-only, and audio-visual adversarial attacks through extensive experiments. What's more, we also propose a novel audio-visual interaction loss (AVIL) for making attackers difficult to find feasible adversarial examples under an allocated attack budget. The loss aims at pushing the inter-class embeddings to be dispersed, namely non-speech and speech clusters, sufficiently disentangled, and pulling the intra-class embeddings as close as possible to keep them compact. Experimental results show the AVIL outperforms the adversarial training by 33.14 mAP (%) under multi-modal attacks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

38 of 52 references.

References · 38

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2016Adversarial examples in the physical worldcited by 1,762
  5. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  6. 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 268
  7. 2019Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognitioncited by 123
  8. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 110
  9. 2018Adversarial Attacks Against Automatic Speech Recognition Systems via Psychoacoustic Hidingcited by 83
  10. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 81
  11. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
  12. 2018Did you hear that? Adversarial Examples Against Automatic Speech Recognitioncited by 48
26 more
  1. 2020SoK: The Faults in our ASRs: An Overview of Attacks against Automatic Speech Recognition and Speaker Identification Systemscited by 46
  2. 2018Robust Audio Adversarial Example for a Physical Attackcited by 42
  3. 2018Targeted Adversarial Examples for Black Box Audio Systemscited by 41
  4. 2020Active Speakers in Contextcited by 40
  5. 2018Fooling End-to-end Speaker Verification by Adversarial Examplescited by 39
  6. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 35
  7. 2021How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wildcited by 30
  8. 2022Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detectioncited by 29
  9. 2021UniCon: Unified Context Network for Robust Active Speaker Detectioncited by 27
  10. 2019Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)cited by 26
  11. 2020Real-time, Universal, and Robust Adversarial Attacks Against Speaker Recognition Systemscited by 26
  12. 2019Personal VAD: Speaker-Conditioned Voice Activity Detectioncited by 24
  13. 2019Adversarial Attacks on Spoofing Countermeasures of automatic speaker verificationcited by 23
  14. 2020Adversarial Attack and Defense Strategies for Deep Speaker Recognition Systemscited by 21
  15. 2020Defense against adversarial attacks on spoofing countermeasures of ASVcited by 20
  16. 2020The Attacker's Perspective on Automatic Speaker Verification: An Overviewcited by 19
  17. 2021Attack on practical speaker verification system using universal adversarial perturbationscited by 17
  18. 2021Adversarial defense for automatic speaker verification by cascaded self-supervised learning modelscited by 16
  19. 2021Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learningcited by 16
  20. 2020Defense for Black-box Attacks on Anti-spoofing Models by Self-Supervised Learningcited by 12
  21. 2021Voting for the right answer: Adversarial defense for speaker verificationcited by 12
  22. 2021Adversarial Sample Detection for Speaker Verification by Neural Vocoderscited by 8
  23. 2021Practical Attacks on Voice Spoofing Countermeasurescited by 7
  24. 2021Learning Spatial-Temporal Graphs for Active Speaker Detectioncited by 4
  25. 2022On Adversarial Robustness of Large-scale Audio Visual Learningcited by 4
  26. 2021Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-based Multimodal Fusioncited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 2.

3 papers.

  1. 2022Multimodal Transformer Distillation for Audio-Visual Synchronizationcited by 2
  2. 2025Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modelingcited by 2
  3. 2025Towards Generalized Source Tracing for Codec-Based Deepfake Speechcited by 0