# ADAPTERMIX: Exploring the Efficacy of Mixture of Adapters for Low-Resource TTS Adaptation

arXiv: https://arxiv.org/abs/2305.18028 · HTML: https://mlsanity.com/paper/2305.18028/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021)
- [AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models](https://mlsanity.com/paper/2302.07027/) (2023)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [Deep Speaker: an End-to-End Neural Speaker Embedding System](https://mlsanity.com/paper/1705.02304/) (2017)
- [A Review of Deep Learning Techniques for Speech Processing](https://mlsanity.com/paper/2305.00359/) (2023)
- [Sample Efficient Adaptive Text-to-Speech](https://mlsanity.com/paper/1809.10460/) (2018)
- [AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios](https://mlsanity.com/paper/2204.00436/) (2022)
- [Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech](https://mlsanity.com/paper/2111.04040/) (2021)
- [Expressive Text-to-Speech using Style Tag](https://mlsanity.com/paper/2104.00436/) (2021)
- [Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis](https://mlsanity.com/paper/2011.06465/) (2020)
- [Multilingual Machine Translation with Hyper-Adapters](https://mlsanity.com/paper/2205.10835/) (2022)
- [Adapting TTS models For New Speakers using Transfer Learning](https://mlsanity.com/paper/2110.05798/) (2021)
- [High quality, lightweight and adaptable TTS using LPCNet](https://mlsanity.com/paper/1905.00590/) (2019)
- [Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptation](https://mlsanity.com/paper/2210.15868/) (2022)
- [Hierarchical prosody modeling and control in non-autoregressive parallel neural TTS](https://mlsanity.com/paper/2110.02952/) (2021)
- [Continual Speaker Adaptation for Text-to-Speech Synthesis](https://mlsanity.com/paper/2103.14512/) (2021)
- [TDASS: Target Domain Adaptation Speech Synthesis Framework for Multi-speaker Low-Resource TTS](https://mlsanity.com/paper/2205.11824/) (2022)
- [MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answering](https://mlsanity.com/paper/2303.01239/) (2023)

## Built on by (most cited first, 7 of 7)

- [Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters](https://mlsanity.com/paper/2402.00828/) (2024)
- [Lightweight Zero-shot Text-to-Speech with Mixture of Adapters](https://mlsanity.com/paper/2407.01291/) (2024)
- [Improving Recursive Transformers with Mixture of LoRAs](https://mlsanity.com/paper/2512.12880/) (2025)
- [HyperTTS: Parameter Efficient Adaptation in Text to Speech using Hypernetworks](https://mlsanity.com/paper/2404.04645/) (2024)
- [Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptation](https://mlsanity.com/paper/2406.17257/) (2024)
- [Unseen Speaker and Language Adaptation for Lightweight Text-To-Speech with Adapters](https://mlsanity.com/paper/2508.18006/) (2025)
- [ZeSTA: Zero-Shot TTS Augmentation with Domain-Conditioned Training for Data-Efficient Personalized Speech Synthesis](https://mlsanity.com/paper/2603.04219/) (2026)

## Abstract (arXiv)

There are significant challenges for speaker adaptation in text-to-speech for languages that are not widely spoken or for speakers with accents or dialects that are not well-represented in the training data. To address this issue, we propose the use of the "mixture of adapters" method. This approach involves adding multiple adapters within a backbone-model layer to learn the unique characteristics of different speakers. Our approach outperforms the baseline, with a noticeable improvement of 5% observed in speaker preference tests when using only one minute of data for each new speaker. Moreover, following the adapter paradigm, we fine-tune only the adapter parameters (11% of the total model parameters). This is a significant achievement in parameter-efficient speaker adaptation, and one of the first models of its kind. Overall, our proposed approach offers a promising solution to the speech synthesis techniques, particularly for adapting to speakers from diverse backgrounds.