# Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models

arXiv: https://arxiv.org/abs/2206.02246 · HTML: https://mlsanity.com/paper/2206.02246/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 5 Jun 2022.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 19 of 19)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversion](https://mlsanity.com/paper/1907.12279/) (2019)
- [Improving Zero-shot Voice Style Transfer via Disentangled Representation Learning](https://mlsanity.com/paper/2103.09420/) (2021)
- [TTS Skins: Speaker Conversion via ASR](https://mlsanity.com/paper/1904.08983/) (2019)
- [VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamics](https://mlsanity.com/paper/2010.02977/) (2020)
- [DGC-vector: A new speaker embedding for zero-shot voice conversion](https://mlsanity.com/paper/2203.09722/) (2022)
- [StarGAN-ZSVC: Towards Zero-Shot Voice Conversion in Low-Resource Contexts](https://mlsanity.com/paper/2106.00043/) (2021)

## Built on by (most cited first, 20 of 20)

- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [PTQD: Accurate Post-Training Quantization for Diffusion Models](https://mlsanity.com/paper/2305.10657/) (2023)
- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregation](https://mlsanity.com/paper/2303.11579/) (2023)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [Speech Driven Video Editing via an Audio-Conditioned Diffusion Model](https://mlsanity.com/paper/2301.04474/) (2023)
- [Federated Generative Learning with Foundation Models](https://mlsanity.com/paper/2306.16064/) (2023)
- [DiffTAD: Temporal Action Detection with Proposal Denoising Diffusion](https://mlsanity.com/paper/2303.14863/) (2023)
- [HiFTNet: A Fast High-Quality Neural Vocoder with Harmonic-plus-Noise Filter and Inverse Short Time Fourier Transform](https://mlsanity.com/paper/2309.09493/) (2023)
- [Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding](https://mlsanity.com/paper/2307.15484/) (2023)
- [OCD: Learning to Overfit with Conditional Diffusion Models](https://mlsanity.com/paper/2210.00471/) (2022)
- [VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing](https://mlsanity.com/paper/2408.05758/) (2024)
- [High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models](https://mlsanity.com/paper/2309.15512/) (2023)
- [PQD: Post-training Quantization for Efficient Diffusion Models](https://mlsanity.com/paper/2501.00124/) (2024)
- [CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators](https://mlsanity.com/paper/2407.06141/) (2024)
- [SLMGAN: Exploiting Speech Language Model Representations for Unsupervised Zero-Shot Voice Conversion in GANs](https://mlsanity.com/paper/2307.09435/) (2023)
- [Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencoder](https://mlsanity.com/paper/2212.08329/) (2022)

## Abstract (arXiv)

We present a novel way of conditioning a pretrained denoising diffusion speech model to produce speech in the voice of a novel person unseen during training. The method requires a short (~3 seconds) sample from the target person, and generation is steered at inference time, without any training steps. At the heart of the method lies a sampling process that combines the estimation of the denoising model with a low-pass version of the new speaker's sample. The objective and subjective evaluations show that our sampling method can generate a voice similar to that of the target speaker in terms of frequency, with an accuracy comparable to state-of-the-art methods, and without training.