# Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech

arXiv: https://arxiv.org/abs/2111.04040 · HTML: https://mlsanity.com/paper/2111.04040/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 7 Nov 2021.

## Builds on (most cited first, 28 of 28)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://mlsanity.com/paper/1710.07654/) (2017)
- [Sample Efficient Adaptive Text-to-Speech](https://mlsanity.com/paper/1809.10460/) (2018)
- [MBNet: MOS Prediction for Synthesized Speech with Mean-Bias Network](https://mlsanity.com/paper/2103.00110/) (2021)
- [Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech](https://mlsanity.com/paper/2103.04088/) (2021)
- [Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding](https://mlsanity.com/paper/2005.08484/) (2020)
- [Utilizing Self-supervised Representations for MOS Prediction](https://mlsanity.com/paper/2104.03017/) (2021)
- [Revisiting Meta-Learning as Supervised Learning](https://mlsanity.com/paper/2002.00573/) (2020)
- [From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint](https://mlsanity.com/paper/2005.04587/) (2020)

## Built on by (most cited first, 22 of 22)

- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias](https://mlsanity.com/paper/2306.03509/) (2023)
- [PromptTTS 2: Describing and Generating Voices with Text Prompt](https://mlsanity.com/paper/2309.02285/) (2023)
- [AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios](https://mlsanity.com/paper/2204.00436/) (2022)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech](https://mlsanity.com/paper/2205.07211/) (2022)
- [StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis](https://mlsanity.com/paper/2312.10741/) (2023)
- [Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2308.01831/) (2023)
- [ADAPTERMIX: Exploring the Efficacy of Mixture of Adapters for Low-Resource TTS Adaptation](https://mlsanity.com/paper/2305.18028/) (2023)
- [USAT: A Universal Speaker-Adaptive Text-to-Speech Approach](https://mlsanity.com/paper/2404.18094/) (2024)
- [SF-Speech: Straightened Flow for Zero-Shot Voice Clone](https://mlsanity.com/paper/2410.12399/) (2024)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Self-supervised learning for robust voice cloning](https://mlsanity.com/paper/2204.03421/) (2022)
- [ELF: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis](https://mlsanity.com/paper/2311.11745/) (2023)
- [Meta-Learning in Audio and Speech Processing: An End to End Comprehensive Review](https://mlsanity.com/paper/2408.10330/) (2024)
- [Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selection](https://mlsanity.com/paper/2408.17432/) (2024)
- [SupertonicTTS: Towards Highly Efficient and Streamlined Text-to-Speech System](https://mlsanity.com/paper/2503.23108/) (2025)
- [Few-Shot Cross-Lingual TTS Using Transferable Phoneme Embedding](https://mlsanity.com/paper/2206.15427/) (2022)
- [An analysis on the effects of speaker embedding choice in non auto-regressive TTS](https://mlsanity.com/paper/2307.09898/) (2023)
- [TransPrompt v2: A Transferable Prompting Framework for Cross-task Text Classification](https://mlsanity.com/paper/2308.15010/) (2023)
- [A Comprehensive Survey of Challenges and Opportunities of Few-Shot Learning Across Multiple Domains](https://mlsanity.com/paper/2504.04017/) (2025)
- [ZeSTA: Zero-Shot TTS Augmentation with Domain-Conditioned Training for Data-Efficient Personalized Speech Synthesis](https://mlsanity.com/paper/2603.04219/) (2026)

## Abstract (arXiv)

Personalizing a speech synthesis system is a highly desired application, where the system can generate speech with the user's voice with rare enrolled recordings. There are two main approaches to build such a system in recent works: speaker adaptation and speaker encoding. On the one hand, speaker adaptation methods fine-tune a trained multi-speaker text-to-speech (TTS) model with few enrolled samples. However, they require at least thousands of fine-tuning steps for high-quality adaptation, making it hard to apply on devices. On the other hand, speaker encoding methods encode enrollment utterances into a speaker embedding. The trained TTS model can synthesize the user's speech conditioned on the corresponding speaker embedding. Nevertheless, the speaker encoder suffers from the generalization gap between the seen and unseen speakers. In this paper, we propose applying a meta-learning algorithm to the speaker adaptation method. More specifically, we use Model Agnostic Meta-Learning (MAML) as the training algorithm of a multi-speaker TTS model, which aims to find a great meta-initialization to adapt the model to any few-shot speaker adaptation tasks quickly. Therefore, we can also adapt the meta-trained TTS model to unseen speakers efficiently. Our experiments compare the proposed method (Meta-TTS) with two baselines: a speaker adaptation method baseline and a speaker encoding method baseline. The evaluation results show that Meta-TTS can synthesize high speaker-similarity speech from few enrollment samples with fewer adaptation steps than the speaker adaptation baseline and outperforms the speaker encoding baseline under the same training scheme. When the speaker encoder of the baseline is pre-trained with extra 8371 speakers of data, Meta-TTS can still outperform the baseline on LibriTTS dataset and achieve comparable results on VCTK dataset.