mlsanityData to 7 Oct 2026

Neural Voice Cloning with a Few Samples

arXiv 1802.06006 ↗First posted 14 Feb 2018cs.CL cs.LG cs.SDcited by 100

Not on the map. Why?

Abstract (arXiv)

Voice cloning is a highly desired feature for personalized speech interfaces. Neural network based speech synthesis has been shown to generate high quality speech for a large number of speakers. In this paper, we introduce a neural voice cloning system that takes a few audio samples as input. We study two approaches: speaker adaptation and speaker encoding. Speaker adaptation is based on fine-tuning a multi-speaker generative model with a few cloning samples. Speaker encoding is based on training a separate model to directly infer a new speaker embedding from cloning audios and to be used with a multi-speaker generative model. In terms of naturalness of the speech and its similarity to original speaker, both approaches can achieve good performance, even with very few cloning audios. While speaker adaptation can achieve better naturalness and similarity, the cloning time or required memory for the speaker encoding approach is significantly less, making it favorable for low-resource deployment.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 23; 2023: 13; 2024: 17; 2025: 11; 2026: 6.

100 papers.

Sort:

By area:

  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  2. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  3. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  4. 2021A Survey on Neural Speech Synthesiscited by 182
  5. 2021FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Datasetcited by 179
  6. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  7. 2023PatchCraft: Exploring Texture Patch for Efficient AI-generated Image Detectioncited by 119
  8. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  9. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  10. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  11. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  12. 2021Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forwardcited by 82
38 more
  1. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 67
  2. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  3. 2021Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speechcited by 37
  4. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 37
  5. 2022Grad-StyleSpeech: Any-speaker Adaptive Text-to-Speech Synthesis with Diffusion Modelscited by 34
  6. 2021Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectorscited by 32
  7. 2022A Review on Methods and Applications in Multimodal Deep Learningcited by 29
  8. 2021Recent Advances and Trends in Multimodal Deep Learning: A Reviewcited by 26
  9. 2021V2C: Visual Voice Cloningcited by 23
  10. 2021Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speechcited by 22
  11. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  12. 2021GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesiscited by 18
  13. 2023The Ethical Implications of Generative Audio Models: A Systematic Literature Reviewcited by 18
  14. 2021AdaSpeech 2: Adaptive Text to Speech with Untranscribed Datacited by 17
  15. 2021TOHAN: A One-step Approach towards Few-shot Hypothesis Adaptationcited by 17
  16. 2022Human Detection of Political Speech Deepfakes across Transcripts, Audio, and Videocited by 16
  17. 2021Catch-A-Waveform: Learning to Generate Audio from a Single Short Examplecited by 15
  18. 2022Low-Resource Multilingual and Zero-Shot Multispeaker TTScited by 15
  19. 2023ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Modelscited by 14
  20. 2021Speaker Generationcited by 13
  21. 2022Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speechcited by 13
  22. 2022On the Horizon: Interactive and Compositional Deepfakescited by 13
  23. 2021"Hello, It's Me": Deep Learning-based Speech Synthesis Attacks in the Real Worldcited by 12
  24. 2021Adapting TTS models For New Speakers using Transfer Learningcited by 12
  25. 2021Expressive Neural Voice Cloningcited by 11
  26. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  27. 2022Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpuscited by 10
  28. 2022Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptationcited by 10
  29. 2022SpecRNet: Towards Faster and More Accessible Audio DeepFake Detectioncited by 9
  30. 2022Adapter-Based Extension of Multi-Speaker Text-to-Speech Model for New Speakerscited by 9
  31. 2024Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgettingcited by 9
  32. 2025Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbingcited by 9
  33. 2022Voice Filter: Few-shot text-to-speech speaker adaptation using voice conversion as a post-processing modulecited by 8
  34. 2023ADAPTERMIX: Exploring the Efficacy of Mixture of Adapters for Low-Resource TTS Adaptationcited by 7
  35. 2025SoK: How Robust is Audio Watermarking in Generative AI models?cited by 7
  36. 2021Review of end-to-end speech synthesis technology based on deep learningcited by 6
  37. 2021Neural Human Deformation Transfercited by 6
  38. 2022Deepfake: Definitions, Performance Metrics and Standards, Datasets and Benchmarks, and a Meta-Reviewcited by 6

50 most cited of 100 All 100 (JSON)