# Semi-Supervised Learning Based on Reference Model for Low-resource TTS

arXiv: https://arxiv.org/abs/2210.14723 · HTML: https://mlsanity.com/paper/2210.14723/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 25 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Generalization in Generation: A closer look at Exposure Bias](https://mlsanity.com/paper/1910.00292/) (2019)
- [Sample Efficient Adaptive Text-to-Speech](https://mlsanity.com/paper/1809.10460/) (2018)
- [Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis](https://mlsanity.com/paper/1808.10128/) (2018)
- [Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding](https://mlsanity.com/paper/2005.08484/) (2020)
- [AVQVC: One-shot Voice Conversion by Vector Quantization with applying contrastive learning](https://mlsanity.com/paper/2202.10020/) (2022)
- [BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization](https://mlsanity.com/paper/2002.01953/) (2020)
- [AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignment](https://mlsanity.com/paper/2003.01950/) (2020)
- [Semantic Mask for Transformer based End-to-End Speech Recognition](https://mlsanity.com/paper/1912.03010/) (2019)
- [AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN](https://mlsanity.com/paper/2005.05642/) (2020)
- [Teacher-Student Training for Robust Tacotron-based TTS](https://mlsanity.com/paper/1911.02839/) (2019)
- [nnSpeech: Speaker-Guided Conditional Variational Autoencoder for Zero-shot Multi-speaker Text-to-Speech](https://mlsanity.com/paper/2202.10712/) (2022)
- [DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learning](https://mlsanity.com/paper/2202.10976/) (2022)
- [Advancing Momentum Pseudo-Labeling with Conformer and Initialization Strategy](https://mlsanity.com/paper/2110.04948/) (2021)
- [Modeling Multi-speaker Latent Space to Improve Neural TTS: Quick Enrolling New Speaker and Enhancing Premium Voice](https://mlsanity.com/paper/1812.05253/) (2018)
- [TGAVC: Improving Autoencoder Voice Conversion with Text-Guided and Adversarial Training](https://mlsanity.com/paper/2208.04035/) (2022)

## Built on by (most cited first, 2 of 2)

- [Towards Spontaneous Style Modeling with Semi-supervised Pre-training for Conversational Text-to-Speech Synthesis](https://mlsanity.com/paper/2308.16593/) (2023)
- [EM-TTS: Efficiently Trained Low-Resource Mongolian Lightweight Text-to-Speech](https://mlsanity.com/paper/2403.08164/) (2024)

## Abstract (arXiv)

Most previous neural text-to-speech (TTS) methods are mainly based on supervised learning methods, which means they depend on a large training dataset and hard to achieve comparable performance under low-resource conditions. To address this issue, we propose a semi-supervised learning method for neural TTS in which labeled target data is limited, which can also resolve the problem of exposure bias in the previous auto-regressive models. Specifically, we pre-train the reference model based on Fastspeech2 with much source data, fine-tuned on a limited target dataset. Meanwhile, pseudo labels generated by the original reference model are used to guide the fine-tuned model's training further, achieve a regularization effect, and reduce the overfitting of the fine-tuned model during training on the limited target data. Experimental results show that our proposed semi-supervised learning scheme with limited target data significantly improves the voice quality for test data to achieve naturalness and robustness in speech synthesis.