# Fake the Real: Backdoor Attack on Deep Speech Classification via Voice Conversion

arXiv: https://arxiv.org/abs/2306.15875 · HTML: https://mlsanity.com/paper/2306.15875/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Audio jailbreaks and speech backdoors, one of 12 directions in LLM safety and security.
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 28 Jun 2023.

## Placed in

- LLM safety and security / [Audio jailbreaks and speech backdoors](https://mlsanity.com/safety/audio-jailbreaks-and-speech-backdoors/)

## Builds on (most cited first, 23 of 23)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020)
- [Invisible Backdoor Attack with Sample-Specific Triggers](https://mlsanity.com/paper/2012.03816/) (2020)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses](https://mlsanity.com/paper/2012.10544/) (2020)
- [Clean-Label Backdoor Attacks on Video Recognition Models](https://mlsanity.com/paper/2003.03030/) (2020)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning](https://mlsanity.com/paper/2008.03648/) (2020)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [Streaming automatic speech recognition with the transformer model](https://mlsanity.com/paper/2001.02674/) (2020)
- [Pushing the limits of raw waveform speaker recognition](https://mlsanity.com/paper/2203.08488/) (2022)
- [Poison Ink: Robust and Invisible Backdoor Attack](https://mlsanity.com/paper/2108.02488/) (2021)
- [Defending against Model Stealing via Verifying Embedded External Features](https://mlsanity.com/paper/2112.03476/) (2021)
- [Can You Hear It? Backdoor Attacks via Ultrasonic Triggers](https://mlsanity.com/paper/2107.14569/) (2021)
- [Few-Shot Backdoor Attacks on Visual Object Tracking](https://mlsanity.com/paper/2201.13178/) (2022)
- [Going In Style: Audio Backdoors Through Stylistic Transformations](https://mlsanity.com/paper/2211.03117/) (2022)
- [VSVC: Backdoor attack against Keyword Spotting based on Voiceprint Selection and Voice Conversion](https://mlsanity.com/paper/2212.10103/) (2022)
- [Glow-WaveGAN 2: High-quality Zero-shot Text-to-speech Synthesis and Any-to-any Voice Conversion](https://mlsanity.com/paper/2207.01832/) (2022)

## Built on by (most cited first, 10 of 10)

- [Imperceptible Rhythm Backdoor Attacks: Exploring Rhythm Transformation for Embedding Undetectable Vulnerabilities on Speech Recognition](https://mlsanity.com/paper/2406.10932/) (2024)
- [A Backdoor Approach with Inverted Labels Using Dirty Label-Flipping Attacks](https://mlsanity.com/paper/2404.00076/) (2024)
- [Breaking Speaker Recognition with PaddingBack](https://mlsanity.com/paper/2308.04179/) (2023)
- [The Art of Deception: Robust Backdoor Attack using Dynamic Stacking of Triggers](https://mlsanity.com/paper/2401.01537/) (2024)
- [EmoAttack: Utilizing Emotional Voice Conversion for Speech Backdoor Attacks on Deep Speech Classification Models](https://mlsanity.com/paper/2408.15508/) (2024)
- [A Survey of Threats Against Voice Authentication and Anti-Spoofing Systems](https://mlsanity.com/paper/2508.16843/) (2025)
- [EventTrojan: Manipulating Non-Intrusive Speech Quality Assessment via Imperceptible Events](https://mlsanity.com/paper/2309.01480/) (2023)
- [SPBA: Utilizing Speech Large Language Model for Backdoor Attacks on Speech Classification Models](https://mlsanity.com/paper/2506.08346/) (2025)
- [Backdoor Attacks on Speech Emotion Recognition via TTS-Generated Poisoning](https://mlsanity.com/paper/2606.21052/) (2026)
- [Pmeta-TLA: Backdoor Attacks for Speech Classification Models via Meta-Learning with Timbre Leakage Attack](https://mlsanity.com/paper/2607.01702/) (2026)

## Abstract (arXiv)

Deep speech classification has achieved tremendous success and greatly promoted the emergence of many real-world applications. However, backdoor attacks present a new security threat to it, particularly with untrustworthy third-party platforms, as pre-defined triggers set by the attacker can activate the backdoor. Most of the triggers in existing speech backdoor attacks are sample-agnostic, and even if the triggers are designed to be unnoticeable, they can still be audible. This work explores a backdoor attack that utilizes sample-specific triggers based on voice conversion. Specifically, we adopt a pre-trained voice conversion model to generate the trigger, ensuring that the poisoned samples does not introduce any additional audible noise. Extensive experiments on two speech classification tasks demonstrate the effectiveness of our attack. Furthermore, we analyzed the specific scenarios that activated the proposed backdoor and verified its resistance against fine-tuning.