# Towards Stealthy Backdoor Attacks against Speech Recognition via Elements of Sound

arXiv: https://arxiv.org/abs/2307.08208 · HTML: https://mlsanity.com/paper/2307.08208/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Audio jailbreaks and speech backdoors, one of 12 directions in LLM safety and security.
- Builds on 33 of its 85 references that are in the map.
- First posted on arXiv 17 Jul 2023.

## Placed in

- LLM safety and security / [Audio jailbreaks and speech backdoors](https://mlsanity.com/safety/audio-jailbreaks-and-speech-backdoors/)

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020)
- [Invisible Backdoor Attack with Sample-Specific Triggers](https://mlsanity.com/paper/2012.03816/) (2020)
- [Label-Consistent Backdoor Attacks](https://mlsanity.com/paper/1912.02771/) (2019)
- [Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses](https://mlsanity.com/paper/2012.10544/) (2020)
- [Clean-Label Backdoor Attacks on Video Recognition Models](https://mlsanity.com/paper/2003.03030/) (2020)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [Dynamic Backdoor Attacks Against Machine Learning Models](https://mlsanity.com/paper/2003.03675/) (2020)
- [Backdoor Attacks Against Deep Learning Systems in the Physical World](https://mlsanity.com/paper/2006.14580/) (2020)
- [Narcissus: A Practical Clean-Label Backdoor Attack with Limited Information](https://mlsanity.com/paper/2204.05255/) (2022)
- [Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch](https://mlsanity.com/paper/2106.08970/) (2021)
- [SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency](https://mlsanity.com/paper/2302.03251/) (2023)
- [Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protection](https://mlsanity.com/paper/2210.00875/) (2022)
- [BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models](https://mlsanity.com/paper/2110.02467/) (2021)
- [Backdoor Attack in the Physical World](https://mlsanity.com/paper/2104.02361/) (2021)
- [A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks](https://mlsanity.com/paper/2206.08514/) (2022)
- [Backdoor Attack against Speaker Verification](https://mlsanity.com/paper/2010.11607/) (2020)
- [Black-box Dataset Ownership Verification via Backdoor Watermarking](https://mlsanity.com/paper/2209.06015/) (2022)
- [Keyword Transformer: A Self-Attention Model for Keyword Spotting](https://mlsanity.com/paper/2104.00769/) (2021)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [A neural attention model for speech command recognition](https://mlsanity.com/paper/1808.08929/) (2018)
- [Can You Hear It? Backdoor Attacks via Ultrasonic Triggers](https://mlsanity.com/paper/2107.14569/) (2021)
- [Learning Efficient Representations for Keyword Spotting with Triplet Loss](https://mlsanity.com/paper/2101.04792/) (2021)
- [Going In Style: Audio Backdoors Through Stylistic Transformations](https://mlsanity.com/paper/2211.03117/) (2022)
- [UMD: Unsupervised Model Detection for X2X Backdoor Attacks](https://mlsanity.com/paper/2305.18651/) (2023)
- [Defending Against Backdoor Attacks by Layer-wise Feature Analysis](https://mlsanity.com/paper/2302.12758/) (2023)
- [Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2302.14564/) (2023)

## Built on by (most cited first, 30 of 30)

- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Models](https://mlsanity.com/paper/2404.12916/) (2024)
- [REFINE: Inversion-Free Backdoor Defense via Model Reprogramming](https://mlsanity.com/paper/2502.18508/) (2025)
- [PointNCBW: Towards Dataset Ownership Verification for Point Clouds via Negative Clean-label Backdoor Watermark](https://mlsanity.com/paper/2408.05500/) (2024)
- [Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models](https://mlsanity.com/paper/2506.16447/) (2025)
- [Hidden in the Noise: Unveiling Backdoors in Audio LLMs Alignment through Latent Acoustic Pattern Triggers](https://mlsanity.com/paper/2508.02175/) (2025)
- [Invisible Backdoor Attack against Self-supervised Learning](https://mlsanity.com/paper/2405.14672/) (2024)
- [Imperceptible Rhythm Backdoor Attacks: Exploring Rhythm Transformation for Embedding Undetectable Vulnerabilities on Speech Recognition](https://mlsanity.com/paper/2406.10932/) (2024)
- [Neutralizing Backdoors through Information Conflicts for Large Language Models](https://mlsanity.com/paper/2411.18280/) (2024)
- [Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack](https://mlsanity.com/paper/2509.23871/) (2025)
- [Towards Sample-specific Backdoor Attack with Clean Labels via Attribute Trigger](https://mlsanity.com/paper/2312.04584/) (2023)
- [A Backdoor Approach with Inverted Labels Using Dirty Label-Flipping Attacks](https://mlsanity.com/paper/2404.00076/) (2024)
- [Cert-SSBD: Certified Backdoor Defense with Sample-Specific Smoothing Noises](https://mlsanity.com/paper/2504.21730/) (2025)
- [The Art of Deception: Robust Backdoor Attack using Dynamic Stacking of Triggers](https://mlsanity.com/paper/2401.01537/) (2024)
- [CBW: Towards Dataset Ownership Verification for Speaker Verification via Clustering-based Backdoor Watermarking](https://mlsanity.com/paper/2503.05794/) (2025)
- [EmoAttack: Utilizing Emotional Voice Conversion for Speech Backdoor Attacks on Deep Speech Classification Models](https://mlsanity.com/paper/2408.15508/) (2024)
- [Revisiting Training-Inference Trigger Intensity in Backdoor Attacks](https://mlsanity.com/paper/2503.12058/) (2025)
- [Gradient Norm-based Fine-Tuning for Backdoor Defense in Automatic Speech Recognition](https://mlsanity.com/paper/2502.01152/) (2025)
- [Multi-Target Backdoor Attacks Against Speaker Recognition](https://mlsanity.com/paper/2508.08559/) (2025)
- [Where Do Backdoors Live? A Component-Level Analysis of Backdoor Propagation in Speech Language Models](https://mlsanity.com/paper/2510.01157/) (2025)
- [STEP: Detecting Audio Backdoor Attacks via Stability-based Trigger Exposure Profiling](https://mlsanity.com/paper/2603.18103/) (2026)
- [BadHMP: Backdoor Attack against Human Motion Prediction](https://mlsanity.com/paper/2409.19638/) (2024)
- [Adversarial Attacks in Multimodal Systems: A Practitioner's Survey](https://mlsanity.com/paper/2505.03084/) (2025)
- [SPBA: Utilizing Speech Large Language Model for Backdoor Attacks on Speech Classification Models](https://mlsanity.com/paper/2506.08346/) (2025)
- [CertDW: Towards Certified Dataset Ownership Verification via Conformal Calibration](https://mlsanity.com/paper/2506.13160/) (2025)
- [Enhancing All-to-X Backdoor Attacks with Optimized Target Class Mapping](https://mlsanity.com/paper/2511.13356/) (2025)
- [TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecasting](https://mlsanity.com/paper/2605.22365/) (2026)
- [Backdoor Attacks on Speech Emotion Recognition via TTS-Generated Poisoning](https://mlsanity.com/paper/2606.21052/) (2026)
- [Pmeta-TLA: Backdoor Attacks for Speech Classification Models via Meta-Learning with Timbre Leakage Attack](https://mlsanity.com/paper/2607.01702/) (2026)
- [Ouroboros: Self-Referential Backdoor Attacks on Speech Enhancement via Clean Audio Triggers](https://mlsanity.com/paper/2608.30329/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) have been widely and successfully adopted and deployed in various applications of speech recognition. Recently, a few works revealed that these models are vulnerable to backdoor attacks, where the adversaries can implant malicious prediction behaviors into victim models by poisoning their training process. In this paper, we revisit poison-only backdoor attacks against speech recognition. We reveal that existing methods are not stealthy since their trigger patterns are perceptible to humans or machine detection. This limitation is mostly because their trigger patterns are simple noises or separable and distinctive clips. Motivated by these findings, we propose to exploit elements of sound (e.g., pitch and timbre) to design more stealthy yet effective poison-only backdoor attacks. Specifically, we insert a short-duration high-pitched signal as the trigger and increase the pitch of remaining audio clips to `mask' it for designing stealthy pitch-based triggers. We manipulate timbre features of victim audios to design the stealthy timbre-based attack and design a voiceprint selection module to facilitate the multi-backdoor attack. Our attacks can generate more `natural' poisoned samples and therefore are more stealthy. Extensive experiments are conducted on benchmark datasets, which verify the effectiveness of our attacks under different settings (e.g., all-to-one, all-to-all, clean-label, physical, and multi-backdoor settings) and their stealthiness. The code for reproducing main experiments are available at urlhttps://github.com/HanboCai/BadSpeechSoE.