# Universal Acoustic Adversarial Attacks for Flexible Control of Speech-LLMs

arXiv: https://arxiv.org/abs/2505.14286 · HTML: https://mlsanity.com/paper/2505.14286/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 46 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 20 of 20)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Audio Adversarial Examples: Targeted Attacks on Speech-to-Text](https://mlsanity.com/paper/1801.01944/) (2018)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Prompting Large Language Models with Speech Recognition Abilities](https://mlsanity.com/paper/2307.11795/) (2023)
- [Adversarial Attacks Against Automatic Speech Recognition Systems via Psychoacoustic Hiding](https://mlsanity.com/paper/1808.05665/) (2018)
- [Connecting Speech Encoder and Large Language Model for ASR](https://mlsanity.com/paper/2309.13963/) (2023)
- [CommanderSong: A Systematic Approach for Practical Adversarial Voice Recognition](https://mlsanity.com/paper/1801.08535/) (2018)
- [Houdini: Fooling Deep Structured Prediction Models](https://mlsanity.com/paper/1707.05373/) (2017)
- [Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Models](https://mlsanity.com/paper/2410.23861/) (2024)
- [Multilingual and Multi-Accent Jailbreaking of Audio LLMs](https://mlsanity.com/paper/2504.01094/) (2025)
- [Crafting Adversarial Examples For Speech Paralinguistics Applications](https://mlsanity.com/paper/1711.03280/) (2017)
- [ADAGIO: Interactive Experimentation with Adversarial Attack and Defense for Audio](https://mlsanity.com/paper/1805.11852/) (2018)
- [Muting Whisper: A Universal Acoustic Adversarial Attack on Speech Foundation Models](https://mlsanity.com/paper/2405.06134/) (2024)
- [Exploring Targeted Universal Adversarial Perturbations to End-to-end ASR Models](https://mlsanity.com/paper/2104.02757/) (2021)
- ["I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Models](https://mlsanity.com/paper/2502.00718/) (2025)
- [Exploring Finetuned Audio-LLM on Heart Murmur Features](https://mlsanity.com/paper/2501.13884/) (2025)
- [Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment](https://mlsanity.com/paper/2503.19586/) (2025)

## Built on by (most cited first, 6 of 6)

- [SoundBreak: A Systematic Study of Audio-Only Adversarial Attacks on Trimodal Models](https://mlsanity.com/paper/2601.16231/) (2026)
- [Audio Jailbreaks in Large Audio-Language Models: Taxonomy, Attack-Defense Analysis, and Cost-Aware Evaluation](https://mlsanity.com/paper/2605.30031/) (2026)
- [Degrading Voice: A Comprehensive Overview of Robust Voice Conversion Through Input Manipulation](https://mlsanity.com/paper/2512.06304/) (2025)
- [Prosody-driven Jailbreaks in Audio LLMs: A Controlled Study and Mechanistic Analysis](https://mlsanity.com/paper/2607.26541/) (2026)
- [From Inaudible Inputs to Model Failures: Low-Frequency Safety Risks in LALMs](https://mlsanity.com/paper/2608.09158/) (2026)
- [From Acoustic Sensing to Physical Actions: A Comprehensive Survey of Security Risks in Edge-Deployed Audio LLM-Based Agents](https://mlsanity.com/paper/2610.15390/) (2026)

## Abstract (arXiv)

The combination of pre-trained speech encoders with large language models has enabled the development of speech LLMs that can handle a wide range of spoken language processing tasks. While these models are powerful and flexible, this very flexibility may make them more vulnerable to adversarial attacks. To examine the extent of this problem, in this work we investigate universal acoustic adversarial attacks on speech LLMs. Here a fixed, universal, adversarial audio segment is prepended to the original input audio. We initially investigate attacks that cause the model to either produce no output or to perform a modified task overriding the original prompt. We then extend the nature of the attack to be selective so that it activates only when specific input attributes, such as a speaker gender or spoken language, are present. Inputs without the targeted attribute should be unaffected, allowing fine-grained control over the model outputs. Our findings reveal critical vulnerabilities in Qwen2-Audio and Granite-Speech and suggest that similar speech LLMs may be susceptible to universal adversarial attacks. This highlights the need for more robust training strategies and improved resistance to adversarial attacks.