mlsanityData to 7 Oct 2026

Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptation

arXiv 2210.15868 ↗First posted 28 Oct 2022cs.SD cs.CL eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

Adapting a neural text-to-speech (TTS) model to a target speaker typically involves fine-tuning most if not all of the parameters of a pretrained multi-speaker backbone model. However, serving hundreds of fine-tuned neural TTS models is expensive as each of them requires significant footprint and separate computational resources (e.g., accelerators, memory). To scale speaker adapted neural TTS voices to hundreds of speakers while preserving the naturalness and speaker similarity, this paper proposes a parameter-efficient few-shot speaker adaptation, where the backbone model is augmented with trainable lightweight modules called residual adapters. This architecture allows the backbone model to be shared across different target speakers. Experimental results show that the proposed approach can achieve competitive naturalness and speaker similarity compared to the full fine-tuning approaches, while requiring only ~0.1% of the backbone model parameters for each speaker.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 28 Oct 2022.

Builds on

33 of 40 references.

References · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Layer Normalizationcited by 5,579
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  9. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  10. 2017Learning multiple visual domains with residual adapterscited by 553
  11. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  12. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
21 more
  1. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  2. 2018Efficient Neural Audio Synthesiscited by 218
  3. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  4. 2018Neural Voice Cloning with a Few Samplescited by 100
  5. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  6. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 82
  7. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  8. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  9. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  10. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  11. 2018Sample Efficient Adaptive Text-to-Speechcited by 45
  12. 2022AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarioscited by 41
  13. 2021Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speechcited by 30
  14. 2020BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimizationcited by 22
  15. 2021DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021cited by 19
  16. 2021AdaSpeech 2: Adaptive Text to Speech with Untranscribed Datacited by 17
  17. 2020AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIANcited by 14
  18. 2021Adapting TTS models For New Speakers using Transfer Learningcited by 12
  19. 2022Voice Filter: Few-shot text-to-speech speaker adaptation using voice conversion as a post-processing modulecited by 8
  20. 2022A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalizationcited by 8
  21. 2021Continual Speaker Adaptation for Text-to-Speech Synthesiscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 2; 2025: 2.

10 papers.

  1. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  2. 2023Modular Deep Learningcited by 82
  3. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  4. 2023ADAPTERMIX: Exploring the Efficacy of Mixture of Adapters for Low-Resource TTS Adaptationcited by 7
  5. 2024USAT: A Universal Speaker-Adaptive Text-to-Speech Approachcited by 6
  6. 2023Parameter-Efficient Learning for Text-to-Speech Accent Adaptationcited by 4
  7. 2025Voice Cloning: Comprehensive Surveycited by 4
  8. 2023Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech Modelcited by 3
  9. 2024Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptationcited by 1
  10. 2025Unseen Speaker and Language Adaptation for Lightweight Text-To-Speech with Adapterscited by 0