# Defending Your Voice: Adversarial Attack on Voice Conversion

arXiv: https://arxiv.org/abs/2005.08781 · HTML: https://mlsanity.com/paper/2005.08781/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2020.

## Built on by (most cited first, 23 of 23)

- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learning](https://mlsanity.com/paper/2106.00273/) (2021)
- [Voting for the right answer: Adversarial defense for speaker verification](https://mlsanity.com/paper/2106.07868/) (2021)
- ["Hello, It's Me": Deep Learning-based Speech Synthesis Attacks in the Real World](https://mlsanity.com/paper/2109.09598/) (2021)
- [A Benchmark for Multi-speaker Anonymization](https://mlsanity.com/paper/2407.05608/) (2024)
- [Characterizing the adversarial vulnerability of speech self-supervised learning](https://mlsanity.com/paper/2111.04330/) (2021)
- [SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesis](https://mlsanity.com/paper/2504.09839/) (2025)
- [Protecting gender and identity with disentangled speech representations](https://mlsanity.com/paper/2104.11051/) (2021)
- [Generating gender-ambiguous voices for privacy-preserving speech recognition](https://mlsanity.com/paper/2207.01052/) (2022)
- [PhantomSound: Black-Box, Query-Efficient Audio Adversarial Attack via Split-Second Phoneme Injection](https://mlsanity.com/paper/2309.06960/) (2023)
- [SongBsAb: A Dual Prevention Approach against Singing Voice Conversion based Illegal Song Covers](https://mlsanity.com/paper/2401.17133/) (2024)
- [StarGAN-ZSVC: Towards Zero-Shot Voice Conversion in Low-Resource Contexts](https://mlsanity.com/paper/2106.00043/) (2021)
- [Catch You and I Can: Revealing Source Voiceprint Against Voice Conversion](https://mlsanity.com/paper/2302.12434/) (2023)
- [Toward Degradation-Robust Voice Conversion](https://mlsanity.com/paper/2110.07537/) (2021)
- [Lightweight Model Attribution and Detection of Synthetic Speech via Audio Residual Fingerprints](https://mlsanity.com/paper/2411.14013/) (2024)
- [De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attacks](https://mlsanity.com/paper/2507.02606/) (2025)
- [Adversarial Privacy Protection on Speech Enhancement](https://mlsanity.com/paper/2206.08170/) (2022)
- [Exploiting Vulnerabilities in Speech Translation Systems through Targeted Adversarial Attacks](https://mlsanity.com/paper/2503.00957/) (2025)
- [VoiceCloak: A Multi-Dimensional Defense Framework against Unauthorized Diffusion-based Voice Cloning](https://mlsanity.com/paper/2505.12332/) (2025)
- [CloneShield: A Framework for Universal Perturbation Against Zero-Shot Voice Cloning](https://mlsanity.com/paper/2505.19119/) (2025)
- [E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis](https://mlsanity.com/paper/2511.07099/) (2025)
- [Degrading Voice: A Comprehensive Overview of Robust Voice Conversion Through Input Manipulation](https://mlsanity.com/paper/2512.06304/) (2025)
- [VocalBridge: Latent Diffusion-Bridge Purification for Defeating Perturbation-Based Voiceprint Defenses](https://mlsanity.com/paper/2601.02444/) (2026)

## Abstract (arXiv)

Substantial improvements have been achieved in recent years in voice conversion, which converts the speaker characteristics of an utterance into those of another speaker without changing the linguistic content of the utterance. Nonetheless, the improved conversion technologies also led to concerns about privacy and authentication. It thus becomes highly desired to be able to prevent one's voice from being improperly utilized with such voice conversion technologies. This is why we report in this paper the first known attempt to perform adversarial attack on voice conversion. We introduce human imperceptible noise into the utterances of a speaker whose voice is to be defended. Given these adversarial examples, voice conversion models cannot convert other utterances so as to sound like being produced by the defended speaker. Preliminary experiments were conducted on two currently state-of-the-art zero-shot voice conversion models. Objective and subjective evaluation results in both white-box and black-box scenarios are reported. It was shown that the speaker characteristics of the converted utterances were made obviously different from those of the defended speaker, while the adversarial examples of the defended speaker are not distinguishable from the authentic utterances.