mlsanityData to 7 Oct 2026

Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models

arXiv 2206.02246 ↗First posted 5 Jun 2022cs.SD cs.AI cs.LGcited by 20

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We present a novel way of conditioning a pretrained denoising diffusion speech model to produce speech in the voice of a novel person unseen during training. The method requires a short (~3 seconds) sample from the target person, and generation is steered at inference time, without any training steps. At the heart of the method lies a sampling process that combines the estimation of the denoising model with a low-pass version of the new speaker's sample. The objective and subjective evaluations show that our sampling method can generate a voice similar to that of the target speaker in terms of frequency, with an accuracy comparable to state-of-the-art methods, and without training.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 19 of its 25 references that are in the map. First posted on arXiv 5 Jun 2022.

Builds on

19 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  2. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 17

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  4. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  5. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  6. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  7. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  8. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  9. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  10. 2020Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversioncited by 75
  11. 2021StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversioncited by 57
  12. 2019StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversioncited by 37
5 more
  1. 2021Improving Zero-shot Voice Style Transfer via Disentangled Representation Learningcited by 32
  2. 2019TTS Skins: Speaker Conversion via ASRcited by 14
  3. 2020VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamicscited by 14
  4. 2022DGC-vector: A new speaker embedding for zero-shot voice conversioncited by 5
  5. 2021StarGAN-ZSVC: Towards Zero-Shot Voice Conversion in Low-Resource Contextscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 12; 2024: 3.

20 papers.

Sort:

By area:

  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  2. 2023ModelScope Text-to-Video Technical Reportcited by 651
  3. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  4. 2022A Survey on Generative Diffusion Modelcited by 238
  5. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 167
  6. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 108
  7. 2023Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregationcited by 81
  8. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AIcited by 68
  9. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Modelcited by 28
  10. 2023Federated Generative Learning with Foundation Modelscited by 23
  11. 2023DiffTAD: Temporal Action Detection with Proposal Denoising Diffusioncited by 19
  12. 2023Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Codingcited by 10
8 more
  1. 2023HiFTNet: A Fast High-Quality Neural Vocoder with Harmonic-plus-Noise Filter and Inverse Short Time Fourier Transformcited by 10
  2. 2022OCD: Learning to Overfit with Conditional Diffusion Modelscited by 9
  3. 2024VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processingcited by 7
  4. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Modelscited by 6
  5. 2024PQD: Post-training Quantization for Efficient Diffusion Modelscited by 6
  6. 2024CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimatorscited by 4
  7. 2023SLMGAN: Exploiting Speech Language Model Representations for Unsupervised Zero-Shot Voice Conversion in GANscited by 3
  8. 2022Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencodercited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Zero-shot Voice Conversion with Diffusion Transformers
  2. 2025MeanVC: Lightweight and Streaming Zero-Shot Voice Conversion via Mean Flows
  3. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
  4. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models
  5. 2023Single and Few-step Diffusion for Generative Speech Enhancement
  6. 2025Posterior Transition Modeling for Unsupervised Diffusion-Based Speech Enhancement
  7. 2024Improving Robustness of Diffusion-Based Zero-Shot Speech Synthesis via Stable Formant Generation
  8. 2022BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis