# Proactive Detection of Voice Cloning with Localized Watermarking

arXiv: https://arxiv.org/abs/2401.17264 · HTML: https://mlsanity.com/paper/2401.17264/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 74 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 30 of 43)

- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [The Stable Signature: Rooting Watermarks in Latent Diffusion Models](https://mlsanity.com/paper/2303.15435/) (2023)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets](https://mlsanity.com/paper/1903.05662/) (2019)
- [Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data](https://mlsanity.com/paper/2007.08457/) (2020)
- [Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust](https://mlsanity.com/paper/2305.20030/) (2023)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision](https://mlsanity.com/paper/2302.03540/) (2023)
- [Does Audio Deepfake Detection Generalize?](https://mlsanity.com/paper/2203.16263/) (2022)
- [ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild](https://mlsanity.com/paper/2210.02437/) (2022)
- [Audiobox: Unified Audio Generation with Natural Language Prompts](https://mlsanity.com/paper/2312.15821/) (2023)
- [SoundStorm: Efficient Parallel Audio Generation](https://mlsanity.com/paper/2305.09636/) (2023)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)

## Built on by (most cited first, 30 of 78)

- [VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild](https://mlsanity.com/paper/2403.16973/) (2024)
- [Watermark Anything with Localized Messages](https://mlsanity.com/paper/2411.07231/) (2024)
- [SoK: Watermarking for AI-Generated Content](https://mlsanity.com/paper/2411.18479/) (2024)
- [Video Seal: Open and Efficient Video Watermarking](https://mlsanity.com/paper/2412.09492/) (2024)
- [OmniGuard: Hybrid Manipulation Localization via Augmented Versatile Deep Image Watermarking](https://mlsanity.com/paper/2412.01615/) (2024)
- [SynthID-Image: Image watermarking at internet scale](https://mlsanity.com/paper/2510.09263/) (2025)
- [V2A-Mark: Versatile Deep Visual-Audio Watermarking for Manipulation Localization and Copyright Protection](https://mlsanity.com/paper/2404.16824/) (2024)
- [Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling](https://mlsanity.com/paper/2509.08753/) (2025)
- [GS-Hider: Hiding Messages into 3D Gaussian Splatting](https://mlsanity.com/paper/2405.15118/) (2024)
- [AudioMarkBench: Benchmarking Robustness of Audio Watermarking](https://mlsanity.com/paper/2406.06979/) (2024)
- [Images that Sound: Composing Images and Sounds on a Single Canvas](https://mlsanity.com/paper/2405.12221/) (2024)
- [Optimizing Adaptive Attacks against Watermarks for Language Models](https://mlsanity.com/paper/2410.02440/) (2024)
- [Watermarking Autoregressive Image Generation](https://mlsanity.com/paper/2506.16349/) (2025)
- [SecureGS: Boosting the Security and Fidelity of 3D Gaussian Splatting Steganography](https://mlsanity.com/paper/2503.06118/) (2025)
- [A Comprehensive Real-World Assessment of Audio Watermarking Algorithms: Will They Survive Neural Codecs?](https://mlsanity.com/paper/2505.19663/) (2025)
- [Survey on AI-Generated Media Detection: From Non-MLLM to MLLM](https://mlsanity.com/paper/2502.05240/) (2025)
- [Steganalysis on Digital Watermarking: Is Your Defense Truly Impervious?](https://mlsanity.com/paper/2406.09026/) (2024)
- [SoK: How Robust is Audio Watermarking in Generative AI models?](https://mlsanity.com/paper/2503.19176/) (2025)
- [Meta Learning Text-to-Speech Synthesis in over 7000 Languages](https://mlsanity.com/paper/2406.06403/) (2024)
- [SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline](https://mlsanity.com/paper/2505.19314/) (2025)
- [Watermarking Without Standards Is Not AI Governance](https://mlsanity.com/paper/2505.23814/) (2025)
- [VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing](https://mlsanity.com/paper/2404.06674/) (2024)
- [AI-generated Image Detection: Passive or Watermark?](https://mlsanity.com/paper/2411.13553/) (2024)
- [Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation](https://mlsanity.com/paper/2406.02733/) (2024)
- [XAttnMark: Learning Robust Audio Watermarking with Cross-Attention](https://mlsanity.com/paper/2502.04230/) (2025)
- [Guidance Watermarking for Diffusion Models](https://mlsanity.com/paper/2509.22126/) (2025)
- [Audio Codec Augmentation for Robust Collaborative Watermarking of Speech Synthesis](https://mlsanity.com/paper/2409.13382/) (2024)
- [Self Voice Conversion as an Attack against Neural Audio Watermarking](https://mlsanity.com/paper/2601.20432/) (2026)
- [Watermarking for AI Content Detection: A Review on Text, Visual, and Audio Modalities](https://mlsanity.com/paper/2504.03765/) (2025)
- [VoiceMark: Zero-Shot Voice Cloning-Resistant Watermarking Approach Leveraging Speaker-Specific Latents](https://mlsanity.com/paper/2505.21568/) (2025)

## Abstract (arXiv)

In the rapidly evolving field of speech generative models, there is a pressing need to ensure audio authenticity against the risks of voice cloning. We present AudioSeal, the first audio watermarking technique designed specifically for localized detection of AI-generated speech. AudioSeal employs a generator/detector architecture trained jointly with a localization loss to enable localized watermark detection up to the sample level, and a novel perceptual loss inspired by auditory masking, that enables AudioSeal to achieve better imperceptibility. AudioSeal achieves state-of-the-art performance in terms of robustness to real life audio manipulations and imperceptibility based on automatic and human evaluation metrics. Additionally, AudioSeal is designed with a fast, single-pass detector, that significantly surpasses existing models in speed - achieving detection up to two orders of magnitude faster, making it ideal for large-scale and real-time applications.