mlsanityData to 7 Oct 2026

Adapting TTS models For New Speakers using Transfer Learning

arXiv 2110.05798 ↗First posted 12 Oct 2021cs.SD cs.CL eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

Training neural text-to-speech (TTS) models for a new speaker typically requires several hours of high quality speech data. Prior works on voice cloning attempt to address this challenge by adapting pre-trained multi-speaker TTS models for a new voice, using a few minutes of speech data of the new speaker. However, publicly available large multi-speaker datasets are often noisy, thereby resulting in TTS models that are not suitable for use in products. We address this challenge by proposing transfer-learning guidelines for adapting high quality single-speaker TTS models for a new speaker, using only a few minutes of speech data. We conduct an extensive study using different amounts of data for a new speaker and evaluate the synthesized speech in terms of naturalness and voice/style similarity to the target speaker. We find that fine-tuning a single-speaker TTS model on just 30 minutes of data, can yield comparable performance to a model trained from scratch on more than 27 hours of data for both male and female target speakers.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

25 of 33 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  7. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  8. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  9. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  10. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  11. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  12. 2018Neural Voice Cloning with a Few Samplescited by 100
13 more
  1. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  2. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  3. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  4. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 61
  5. 2021Hi-Fi Multi-Speaker English TTS Datasetcited by 60
  6. 2021One TTS Alignment To Rule Them Allcited by 47
  7. 2019Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokenscited by 44
  8. 2017Transfer learning for music classification and regression taskscited by 36
  9. 2021Expressive Neural Voice Cloningcited by 11
  10. 2019Expediting TTS Synthesis with Adversarial Vocodingcited by 10
  11. 2020SpeakerNet: 1D Depth-wise Separable Convolutional Network for Text-Independent Speaker Recognition and Verificationcited by 10
  12. 2021Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learningcited by 9
  13. 2021Multi-task Voice Activated Framework using Self-supervised Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 3; 2025: 1; 2026: 1.

12 papers.

  1. 2022Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Featurescited by 20
  2. 2022Low-Resource Multilingual and Zero-Shot Multispeaker TTScited by 15
  3. 2022Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptationcited by 10
  4. 2022A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTScited by 9
  5. 2023ADAPTERMIX: Exploring the Efficacy of Mixture of Adapters for Low-Resource TTS Adaptationcited by 7
  6. 2024Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settingscited by 6
  7. 2024Boosting keyword spotting through on-device learnable user speech characteristicscited by 4
  8. 2025Voice Cloning: Comprehensive Surveycited by 4
  9. 2024Stable-TTS: Stable Speaker-Adaptive Text-to-Speech Synthesis via Prosody Promptingcited by 3
  10. 2023Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directionscited by 1
  11. 2023Improved Child Text-to-Speech Synthesis through Fastpitch-based Transfer Learningcited by 0
  12. 2026VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptationcited by 0