# Muting Whisper: A Universal Acoustic Adversarial Attack on Speech Foundation Models

arXiv: https://arxiv.org/abs/2405.06134 · HTML: https://mlsanity.com/paper/2405.06134/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Audio Adversarial Examples: Targeted Attacks on Speech-to-Text](https://mlsanity.com/paper/1801.01944/) (2018)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition](https://mlsanity.com/paper/1903.10346/) (2019)
- [Adversarial Attacks Against Automatic Speech Recognition Systems via Psychoacoustic Hiding](https://mlsanity.com/paper/1808.05665/) (2018)
- [DolphinAtack: Inaudible Voice Commands](https://mlsanity.com/paper/1708.09537/) (2017)
- [CommanderSong: A Systematic Approach for Practical Adversarial Voice Recognition](https://mlsanity.com/paper/1801.08535/) (2018)
- [Houdini: Fooling Deep Structured Prediction Models](https://mlsanity.com/paper/1707.05373/) (2017)
- [Did you hear that? Adversarial Examples Against Automatic Speech Recognition](https://mlsanity.com/paper/1801.00554/) (2018)
- [Targeted Adversarial Examples for Black Box Audio Systems](https://mlsanity.com/paper/1805.07820/) (2018)
- [Universal Adversarial Perturbations for Speech Recognition Systems](https://mlsanity.com/paper/1905.03828/) (2019)
- [Practical Hidden Voice Attacks against Speech and Speaker Recognition Systems](https://mlsanity.com/paper/1904.05734/) (2019)
- [SirenAttack: Generating Adversarial Audio for End-to-End Acoustic Systems](https://mlsanity.com/paper/1901.07846/) (2019)
- [Crafting Adversarial Examples For Speech Paralinguistics Applications](https://mlsanity.com/paper/1711.03280/) (2017)
- [Black-box Adversarial Attacks on Commercial Speech Platforms with Minimal Information](https://mlsanity.com/paper/2110.09714/) (2021)
- [ADAGIO: Interactive Experimentation with Adversarial Attack and Defense for Audio](https://mlsanity.com/paper/1805.11852/) (2018)
- [There is more than one kind of robustness: Fooling Whisper with adversarial examples](https://mlsanity.com/paper/2210.17316/) (2022)
- [Exploring Targeted Universal Adversarial Perturbations to End-to-end ASR Models](https://mlsanity.com/paper/2104.02757/) (2021)

## Built on by (most cited first, 12 of 12)

- ["I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Models](https://mlsanity.com/paper/2502.00718/) (2025)
- [Universal Acoustic Adversarial Attacks for Flexible Control of Speech-LLMs](https://mlsanity.com/paper/2505.14286/) (2025)
- [Controlling Whisper: Universal Acoustic Adversarial Attacks to Control Speech Foundation Models](https://mlsanity.com/paper/2407.04482/) (2024)
- [SyncBreaker:Stage-Aware Multimodal Adversarial Attacks on Audio-Driven Talking Head Generation](https://mlsanity.com/paper/2604.08405/) (2026)
- [SoundBreak: A Systematic Study of Audio-Only Adversarial Attacks on Trimodal Models](https://mlsanity.com/paper/2601.16231/) (2026)
- [Codec-Robust Attacks on Audio LLMs](https://mlsanity.com/paper/2605.20519/) (2026)
- [Whisper Smarter, not Harder: Adversarial Attack on Partial Suppression](https://mlsanity.com/paper/2508.09994/) (2025)
- [MORE: Multi-Objective Adversarial Attacks on Speech Recognition](https://mlsanity.com/paper/2601.01852/) (2026)
- [Beyond Waveform Robustness: Robust Feature-Vocoder Adversarial Attacks on Automatic Speech Recognition](https://mlsanity.com/paper/2606.05678/) (2026)
- [Generative Testing of Automated Speech Recognition Systems](https://mlsanity.com/paper/2607.09833/) (2026)
- [From Inaudible Inputs to Model Failures: Low-Frequency Safety Risks in LALMs](https://mlsanity.com/paper/2608.09158/) (2026)
- [Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Models](https://mlsanity.com/paper/2609.27378/) (2026)

## Abstract (arXiv)

Recent developments in large speech foundation models like Whisper have led to their widespread use in many automatic speech recognition (ASR) applications. These systems incorporate `special tokens' in their vocabulary, such as texttt<|endoftext|>, to guide their language generation process. However, we demonstrate that these tokens can be exploited by adversarial attacks to manipulate the model's behavior. We propose a simple yet effective method to learn a universal acoustic realization of Whisper's texttt<|endoftext|> token, which, when prepended to any speech signal, encourages the model to ignore the speech and only transcribe the special token, effectively `muting' the model. Our experiments demonstrate that the same, universal 0.64-second adversarial audio segment can successfully mute a target Whisper ASR model for over 97% of speech samples. Moreover, we find that this universal adversarial audio segment often transfers to new datasets and tasks. Overall this work demonstrates the vulnerability of Whisper models to `muting' adversarial attacks, where such attacks can pose both risks and potential benefits in real-world settings: for example the attack can be used to bypass speech moderation systems, or conversely the attack can also be used to protect private speech data.