# Machine Speech Chain with One-shot Speaker Adaptation

arXiv: https://arxiv.org/abs/1803.10525 · HTML: https://mlsanity.com/paper/1803.10525/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2018.

## Built on by (most cited first, 11 of 11)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation](https://mlsanity.com/paper/2204.02967/) (2022)
- [Injecting Text in Self-Supervised Speech Pretraining](https://mlsanity.com/paper/2108.12226/) (2021)
- [ASR data augmentation in low-resource settings using cross-lingual multi-speaker TTS and cross-lingual voice conversion](https://mlsanity.com/paper/2204.00618/) (2022)
- [A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data](https://mlsanity.com/paper/2506.11130/) (2025)
- [Review of end-to-end speech synthesis technology based on deep learning](https://mlsanity.com/paper/2104.09995/) (2021)
- [Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation](https://mlsanity.com/paper/2104.03815/) (2021)
- [SpeeChain: A Speech Toolkit for Large-Scale Machine Speech Chain](https://mlsanity.com/paper/2301.02966/) (2023)
- [Speaker Adaption with Intuitive Prosodic Features for Statistical Parametric Speech Synthesis](https://mlsanity.com/paper/2203.00951/) (2022)
- [One-shot Machine Teaching: Cost Very Few Examples to Converge Faster](https://mlsanity.com/paper/2212.06416/) (2022)

## Abstract (arXiv)

In previous work, we developed a closed-loop speech chain model based on deep learning, in which the architecture enabled the automatic speech recognition (ASR) and text-to-speech synthesis (TTS) components to mutually improve their performance. This was accomplished by the two parts teaching each other using both labeled and unlabeled data. This approach could significantly improve model performance within a single-speaker speech dataset, but only a slight increase could be gained in multi-speaker tasks. Furthermore, the model is still unable to handle unseen speakers. In this paper, we present a new speech chain mechanism by integrating a speaker recognition model inside the loop. We also propose extending the capability of TTS to handle unseen speakers by implementing one-shot speaker adaptation. This enables TTS to mimic voice characteristics from one speaker to another with only a one-shot speaker sample, even from a text without any speaker information. In the speech chain loop mechanism, ASR also benefits from the ability to further learn an arbitrary speaker's characteristics from the generated speech waveform, resulting in a significant improvement in the recognition rate.