# Mitigating Unauthorized Speech Synthesis for Voice Protection

arXiv: https://arxiv.org/abs/2410.20742 · HTML: https://mlsanity.com/paper/2410.20742/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [A Survey on Neural Network Interpretability](https://mlsanity.com/paper/2012.14261/) (2020)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?](https://mlsanity.com/paper/1706.00712/) (2017)
- [Unlearnable Examples: Making Personal Data Unexploitable](https://mlsanity.com/paper/2101.04898/) (2021)
- [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021)
- [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://mlsanity.com/paper/1710.08969/) (2017)
- [Speaker Anonymization Using X-vector and Neural Waveform Models](https://mlsanity.com/paper/1905.13561/) (2019)
- [Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2307.07218/) (2023)
- [Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compression](https://mlsanity.com/paper/2301.13838/) (2023)
- [DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations](https://mlsanity.com/paper/2103.02079/) (2021)
- [Not All Poisons are Created Equal: Robust Training against Data Poisoning](https://mlsanity.com/paper/2210.09671/) (2022)
- [VSMask: Defending Against Voice Synthesis Attack via Real-Time Predictive Perturbation](https://mlsanity.com/paper/2305.05736/) (2023)
- [Voice-Indistinguishability: Protecting Voiceprint in Privacy-Preserving Speech Data Release](https://mlsanity.com/paper/2004.07442/) (2020)
- [Distinctive and Natural Speaker Anonymization via Singular Value Transformation-assisted Matrix](https://mlsanity.com/paper/2405.10786/) (2024)

## Built on by (most cited first, 8 of 8)

- [AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Models](https://mlsanity.com/paper/2505.14103/) (2025)
- [SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesis](https://mlsanity.com/paper/2504.09839/) (2025)
- [A Survey on Unlearnable Data](https://mlsanity.com/paper/2503.23536/) (2025)
- [To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model](https://mlsanity.com/paper/2605.14291/) (2026)
- [Off-The-Shelf Image-to-Image Models Are All You Need To Defeat Image Protection Schemes](https://mlsanity.com/paper/2602.22197/) (2026)
- [Boosting the Transferability of Audio Adversarial Examples with Acoustic Representation Optimization](https://mlsanity.com/paper/2503.19591/) (2025)
- [E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis](https://mlsanity.com/paper/2511.07099/) (2025)
- [VocalBridge: Latent Diffusion-Bridge Purification for Defeating Perturbation-Based Voiceprint Defenses](https://mlsanity.com/paper/2601.02444/) (2026)

## Abstract (arXiv)

With just a few speech samples, it is possible to perfectly replicate a speaker's voice in recent years, while malicious voice exploitation (e.g., telecom fraud for illegal financial gain) has brought huge hazards in our daily lives. Therefore, it is crucial to protect publicly accessible speech data that contains sensitive information, such as personal voiceprints. Most previous defense methods have focused on spoofing speaker verification systems in timbre similarity but the synthesized deepfake speech is still of high quality. In response to the rising hazards, we devise an effective, transferable, and robust proactive protection technology named Pivotal Objective Perturbation (POP) that applies imperceptible error-minimizing noises on original speech samples to prevent them from being effectively learned for text-to-speech (TTS) synthesis models so that high-quality deepfake speeches cannot be generated. We conduct extensive experiments on state-of-the-art (SOTA) TTS models utilizing objective and subjective metrics to comprehensively evaluate our proposed method. The experimental results demonstrate outstanding effectiveness and transferability across various models. Compared to the speech unclarity score of 21.94% from voice synthesizers trained on samples without protection, POP-protected samples significantly increase it to 127.31%. Moreover, our method shows robustness against noise reduction and data augmentation techniques, thereby greatly reducing potential hazards.