# Speech Watermarking with Discrete Intermediate Representations

arXiv: https://arxiv.org/abs/2412.13917 · HTML: https://mlsanity.com/paper/2412.13917/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 54 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 21 of 21)

- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs](https://mlsanity.com/paper/2407.04051/) (2024)
- [Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation](https://mlsanity.com/paper/2202.12233/) (2022)
- [Latent Video Transformer](https://mlsanity.com/paper/2006.10704/) (2020)
- [InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt](https://mlsanity.com/paper/2301.13662/) (2023)
- [WavChat: A Survey of Spoken Dialogue Models](https://mlsanity.com/paper/2411.13577/) (2024)
- [SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization](https://mlsanity.com/paper/2205.07547/) (2022)
- [Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias](https://mlsanity.com/paper/2306.03509/) (2023)
- [WavMark: Watermarking for Audio Generation](https://mlsanity.com/paper/2308.12770/) (2023)
- [Detecting Voice Cloning Attacks via Timbre Watermarking](https://mlsanity.com/paper/2312.03410/) (2023)
- [Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2307.07218/) (2023)
- [VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature](https://mlsanity.com/paper/2204.00768/) (2022)
- [ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control](https://mlsanity.com/paper/2406.01205/) (2024)
- [MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2402.09378/) (2024)
- [Language-Codec: Bridging Discrete Codec Representations and Speech Language Models](https://mlsanity.com/paper/2402.12208/) (2024)

## Abstract (arXiv)

Speech watermarking techniques can proactively mitigate the potential harmful consequences of instant voice cloning techniques. These techniques involve the insertion of signals into speech that are imperceptible to humans but can be detected by algorithms. Previous approaches typically embed watermark messages into continuous space. However, intuitively, embedding watermark information into robust discrete latent space can significantly improve the robustness of watermarking systems. In this paper, we propose DiscreteWM, a novel speech watermarking framework that injects watermarks into the discrete intermediate representations of speech. Specifically, we map speech into discrete latent space with a vector-quantized autoencoder and inject watermarks by changing the modular arithmetic relation of discrete IDs. To ensure the imperceptibility of watermarks, we also propose a manipulator model to select the candidate tokens for watermark embedding. Experimental results demonstrate that our framework achieves state-of-the-art performance in robustness and imperceptibility, simultaneously. Moreover, our flexible frame-wise approach can serve as an efficient solution for both voice cloning detection and information hiding. Additionally, DiscreteWM can encode 1 to 150 bits of watermark information within a 1-second speech clip, indicating its encoding capacity. Audio samples are available at https://DiscreteWM.github.io/discrete_wm.