# Hidden in the Noise: Unveiling Backdoors in Audio LLMs Alignment through Latent Acoustic Pattern Triggers

arXiv: https://arxiv.org/abs/2508.02175 · HTML: https://mlsanity.com/paper/2508.02175/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Audio jailbreaks and speech backdoors, one of 12 directions in LLM safety and security.
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Placed in

- LLM safety and security / [Audio jailbreaks and speech backdoors](https://mlsanity.com/safety/audio-jailbreaks-and-speech-backdoors/)

## Builds on (most cited first, 24 of 24)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2007.02343/) (2020)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements](https://mlsanity.com/paper/2006.01043/) (2020)
- [Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming](https://mlsanity.com/paper/2408.16725/) (2024)
- [Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents](https://mlsanity.com/paper/2402.11208/) (2024)
- [Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation](https://mlsanity.com/paper/2403.09572/) (2024)
- [Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review](https://mlsanity.com/paper/2007.10760/) (2020)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch](https://mlsanity.com/paper/2106.08970/) (2021)
- [VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models](https://mlsanity.com/paper/2402.13851/) (2024)
- [On the Role of Attention Heads in Large Language Model Safety](https://mlsanity.com/paper/2410.13708/) (2024)
- [Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models](https://mlsanity.com/paper/2210.09545/) (2022)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Can You Hear It? Backdoor Attacks via Ultrasonic Triggers](https://mlsanity.com/paper/2107.14569/) (2021)
- [Towards Stealthy Backdoor Attacks against Speech Recognition via Elements of Sound](https://mlsanity.com/paper/2307.08208/) (2023)
- [Speech Translation with Large Language Models: An Industrial Practice](https://mlsanity.com/paper/2312.13585/) (2023)
- [A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluations](https://mlsanity.com/paper/2502.05224/) (2025)
- [A Unified Framework for Adversarial Attack and Defense in Constrained Feature Space](https://mlsanity.com/paper/2112.01156/) (2021)
- [Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents](https://mlsanity.com/paper/2505.14418/) (2025)
- [Modeling Spatio-temporal Dynamical Systems with Neural Discrete Learning and Levels-of-Experts](https://mlsanity.com/paper/2402.05970/) (2024)
- [Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant](https://mlsanity.com/paper/2410.15316/) (2024)
- [UniErase: Towards Balanced and Precise Unlearning in Language Models](https://mlsanity.com/paper/2505.15674/) (2025)

## Built on by (most cited first, 8 of 8)

- [PointCoT: A Multi-modal Benchmark for Explicit 3D Geometric Reasoning](https://mlsanity.com/paper/2602.23945/) (2026)
- [Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Models](https://mlsanity.com/paper/2510.10265/) (2025)
- [Audio Jailbreaks in Large Audio-Language Models: Taxonomy, Attack-Defense Analysis, and Cost-Aware Evaluation](https://mlsanity.com/paper/2605.30031/) (2026)
- [Sparsity-Aware Voxel Attention and Foreground Modulation for 3D Semantic Scene Completion](https://mlsanity.com/paper/2604.05780/) (2026)
- [EchoDistill: Robust Large Audio Language Models via Noisy-to-Clean Self-Distillation](https://mlsanity.com/paper/2605.23954/) (2026)
- [Piggybacking on Perception: Stealthy Concurrent Audio Prompt Injections against Multimodal LLM Agents](https://mlsanity.com/paper/2607.28165/) (2026)
- [From Inaudible Inputs to Model Failures: Low-Frequency Safety Risks in LALMs](https://mlsanity.com/paper/2608.09158/) (2026)
- [From Acoustic Sensing to Physical Actions: A Comprehensive Survey of Security Risks in Edge-Deployed Audio LLM-Based Agents](https://mlsanity.com/paper/2610.15390/) (2026)

## Abstract (arXiv)

As Audio Large Language Models (ALLMs) emerge as powerful tools for speech processing, their safety implications demand urgent attention. While considerable research has explored textual and vision safety, audio's distinct characteristics present significant challenges. This paper first investigates: Is ALLM vulnerable to backdoor attacks exploiting acoustic triggers? In response to this issue, we introduce Hidden in the Noise (HIN), a novel backdoor attack framework designed to exploit subtle, audio-specific features. HIN applies acoustic modifications to raw audio waveforms, such as alterations to temporal dynamics and strategic injection of spectrally tailored noise. These changes introduce consistent patterns that an ALLM's acoustic feature encoder captures, embedding robust triggers within the audio stream. To evaluate ALLM robustness against audio-feature-based triggers, we develop the AudioSafe benchmark, assessing nine distinct risk types. Extensive experiments on AudioSafe and three established safety datasets reveal critical vulnerabilities in existing ALLMs: (I) audio features like environment noise and speech rate variations achieve over 90% average attack success rate. (II) ALLMs exhibit significant sensitivity differences across acoustic features, particularly showing minimal response to volume as a trigger, and (III) poisoned sample inclusion causes only marginal loss curve fluctuations, highlighting the attack's stealth.