# Erasing Your Voice Before It's Heard: Training-free Speaker Unlearning for Zero-shot Text-to-Speech

arXiv: https://arxiv.org/abs/2601.20481 · HTML: https://mlsanity.com/paper/2601.20481/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2026.

## Built on by (most cited first, 4 of 4)

- [Speech Generation Speaker Poisoning: Capability Erasure in Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2603.07551/) (2026)
- [Forget who you Forgot: Speaker Unlearning to Prevent Re-Identification in Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2609.27399/) (2026)
- [Inference-Time Target Speaker Unlearning in LLM-Based Automatic Speech Recognition](https://mlsanity.com/paper/2609.30439/) (2026)
- [Evaluating Machine Unlearning in ASR](https://mlsanity.com/paper/2609.34092/) (2026)

## Abstract (arXiv)

Modern zero-shot text-to-speech (TTS) models offer unprecedented expressivity but also pose serious crime risks, as they can synthesize voices of individuals who never consented. In this context, speaker unlearning aims to prevent the generation of specific speaker identities upon request. Existing approaches, reliant on retraining, are costly and limited to speakers seen in the training set. We present TruS, a training-free speaker unlearning framework that shifts the paradigm from data deletion to inference-time control. TruS steers identity-specific hidden activations to suppress target speakers while preserving other attributes (e.g., prosody and emotion). Experimental results show that TruS effectively prevents voice generation on both seen and unseen opt-out speakers, establishing a scalable safeguard for speech synthesis. The demo and code are available on http://mmai.ewha.ac.kr/trus.