# Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection

arXiv: https://arxiv.org/abs/2210.00753 · HTML: https://mlsanity.com/paper/2210.00753/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 30 of 38)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Audio Adversarial Examples: Targeted Attacks on Speech-to-Text](https://mlsanity.com/paper/1801.01944/) (2018)
- [Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition](https://mlsanity.com/paper/1903.10346/) (2019)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [Adversarial Attacks Against Automatic Speech Recognition Systems via Psychoacoustic Hiding](https://mlsanity.com/paper/1808.05665/) (2018)
- [AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection](https://mlsanity.com/paper/1901.01342/) (2019)
- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [Did you hear that? Adversarial Examples Against Automatic Speech Recognition](https://mlsanity.com/paper/1801.00554/) (2018)
- [SoK: The Faults in our ASRs: An Overview of Attacks against Automatic Speech Recognition and Speaker Identification Systems](https://mlsanity.com/paper/2007.06622/) (2020)
- [Robust Audio Adversarial Example for a Physical Attack](https://mlsanity.com/paper/1810.11793/) (2018)
- [Targeted Adversarial Examples for Black Box Audio Systems](https://mlsanity.com/paper/1805.07820/) (2018)
- [Active Speakers in Context](https://mlsanity.com/paper/2005.09812/) (2020)
- [Fooling End-to-end Speaker Verification by Adversarial Examples](https://mlsanity.com/paper/1801.03339/) (2018)
- [MAAS: Multi-modal Assignation for Active Speaker Detection](https://mlsanity.com/paper/2101.03682/) (2021)
- [How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild](https://mlsanity.com/paper/2106.03932/) (2021)
- [Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2207.07783/) (2022)
- [UniCon: Unified Context Network for Robust Active Speaker Detection](https://mlsanity.com/paper/2108.02607/) (2021)
- [Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)](https://mlsanity.com/paper/1906.10555/) (2019)
- [Real-time, Universal, and Robust Adversarial Attacks Against Speaker Recognition Systems](https://mlsanity.com/paper/2003.02301/) (2020)
- [Personal VAD: Speaker-Conditioned Voice Activity Detection](https://mlsanity.com/paper/1908.04284/) (2019)
- [Adversarial Attacks on Spoofing Countermeasures of automatic speaker verification](https://mlsanity.com/paper/1910.08716/) (2019)
- [Adversarial Attack and Defense Strategies for Deep Speaker Recognition Systems](https://mlsanity.com/paper/2008.07685/) (2020)
- [Defense against adversarial attacks on spoofing countermeasures of ASV](https://mlsanity.com/paper/2003.03065/) (2020)
- [The Attacker's Perspective on Automatic Speaker Verification: An Overview](https://mlsanity.com/paper/2004.08849/) (2020)
- [Attack on practical speaker verification system using universal adversarial perturbations](https://mlsanity.com/paper/2105.09022/) (2021)
- [Adversarial defense for automatic speaker verification by cascaded self-supervised learning models](https://mlsanity.com/paper/2102.07047/) (2021)

## Built on by (most cited first, 3 of 3)

- [Multimodal Transformer Distillation for Audio-Visual Synchronization](https://mlsanity.com/paper/2210.15563/) (2022)
- [Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling](https://mlsanity.com/paper/2508.02000/) (2025)
- [Towards Generalized Source Tracing for Codec-Based Deepfake Speech](https://mlsanity.com/paper/2506.07294/) (2025)

## Abstract (arXiv)

Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications. However, to the best of our knowledge, the adversarial robustness of AVASD models hasn't been investigated, not to mention the effective defense against such attacks. In this paper, we are the first to reveal the vulnerability of AVASD models under audio-only, visual-only, and audio-visual adversarial attacks through extensive experiments. What's more, we also propose a novel audio-visual interaction loss (AVIL) for making attackers difficult to find feasible adversarial examples under an allocated attack budget. The loss aims at pushing the inter-class embeddings to be dispersed, namely non-speech and speech clusters, sufficiently disentangled, and pulling the intra-class embeddings as close as possible to keep them compact. Experimental results show the AVIL outperforms the adversarial training by 33.14 mAP (%) under multi-modal attacks.