# CloneShield: A Framework for Universal Perturbation Against Zero-Shot Voice Cloning

arXiv: https://arxiv.org/abs/2505.19119 · HTML: https://mlsanity.com/paper/2505.19119/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 70 references that are in the map.
- First posted on arXiv 25 May 2025.

## Builds on (most cited first, 30 of 36)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Unlearnable Examples: Making Personal Data Unexploitable](https://mlsanity.com/paper/2101.04898/) (2021)
- [XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model](https://mlsanity.com/paper/2406.04904/) (2024)
- [Diff-TTS: A Denoising Diffusion Model for Text-to-Speech](https://mlsanity.com/paper/2104.01409/) (2021)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [Matcha-TTS: A fast TTS architecture with conditional flow matching](https://mlsanity.com/paper/2309.03199/) (2023)
- [Powerset multi-class cross entropy loss for neural speaker diarization](https://mlsanity.com/paper/2310.13025/) (2023)
- [LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus](https://mlsanity.com/paper/2305.18802/) (2023)
- [MatterGen: a generative model for inorganic materials design](https://mlsanity.com/paper/2312.03687/) (2023)
- [The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech](https://mlsanity.com/paper/2409.09305/) (2024)
- [Proactive Detection of Voice Cloning with Localized Watermarking](https://mlsanity.com/paper/2401.17264/) (2024)
- [BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data](https://mlsanity.com/paper/2402.08093/) (2024)
- [Detecting Voice Cloning Attacks via Timbre Watermarking](https://mlsanity.com/paper/2312.03410/) (2023)
- [Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling](https://mlsanity.com/paper/2103.14574/) (2021)
- [IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System](https://mlsanity.com/paper/2502.05512/) (2025)
- [The Audio Auditor: User-Level Membership Inference in Internet of Things Voice Services](https://mlsanity.com/paper/1905.07082/) (2019)
- [V2C: Visual Voice Cloning](https://mlsanity.com/paper/2111.12890/) (2021)
- [Defending Your Voice: Adversarial Attack on Voice Conversion](https://mlsanity.com/paper/2005.08781/) (2020)

## Built on by (most cited first, 1 of 1)

- [Degrading Voice: A Comprehensive Overview of Robust Voice Conversion Through Input Manipulation](https://mlsanity.com/paper/2512.06304/) (2025)

## Abstract (arXiv)

Recent breakthroughs in text-to-speech (TTS) voice cloning have raised serious privacy concerns, allowing highly accurate vocal identity replication from just a few seconds of reference audio, while retaining the speaker's vocal authenticity. In this paper, we introduce CloneShield, a universal time-domain adversarial perturbation framework specifically designed to defend against zero-shot voice cloning. Our method provides protection that is robust across speakers and utterances, without requiring any prior knowledge of the synthesized text. We formulate perturbation generation as a multi-objective optimization problem, and propose Multi-Gradient Descent Algorithm (MGDA) to ensure the robust protection across diverse utterances. To preserve natural auditory perception for users, we decompose the adversarial perturbation via Mel-spectrogram representations and fine-tune it for each sample. This design ensures imperceptibility while maintaining strong degradation effects on zero-shot cloned outputs. Experiments on three state-of-the-art zero-shot TTS systems, five benchmark datasets and evaluations from 60 human listeners demonstrate that our method preserves near-original audio quality in protected inputs (PESQ = 3.90, SRS = 0.93) while substantially degrading both speaker similarity and speech quality in cloned samples (PESQ = 1.07, SRS = 0.08).