# Simple and Effective Unsupervised Speech Synthesis

arXiv: https://arxiv.org/abs/2204.02524 · HTML: https://mlsanity.com/paper/2204.02524/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://mlsanity.com/paper/1710.08969/) (2017)
- [PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS](https://mlsanity.com/paper/2103.15060/) (2021)
- [Listening while Speaking: Speech Chain by Deep Learning](https://mlsanity.com/paper/1707.04879/) (2017)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [Almost Unsupervised Text to Speech and Automatic Speech Recognition](https://mlsanity.com/paper/1905.06791/) (2019)
- [Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis](https://mlsanity.com/paper/1808.10128/) (2018)
- [Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning](https://mlsanity.com/paper/1910.12729/) (2019)
- [ECAPA-TDNN for Multi-speaker Text-to-speech Synthesis](https://mlsanity.com/paper/2203.10473/) (2022)

## Built on by (most cited first, 12 of 12)

- [Simple and Effective Unsupervised Speech Translation](https://mlsanity.com/paper/2210.10191/) (2022)
- [Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE](https://mlsanity.com/paper/2206.02512/) (2022)
- [Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speech](https://mlsanity.com/paper/2303.11131/) (2023)
- [Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation](https://mlsanity.com/paper/2305.07455/) (2023)
- [Unsupervised Speech Recognition with N-Skipgram and Positional Unigram Matching](https://mlsanity.com/paper/2310.02382/) (2023)
- [Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis](https://mlsanity.com/paper/2404.19622/) (2024)
- [ParrotTTS: Text-to-Speech synthesis by exploiting self-supervised representations](https://mlsanity.com/paper/2303.01261/) (2023)
- [Learning the joint distribution of two sequences using little or no paired data](https://mlsanity.com/paper/2212.03232/) (2022)
- [Textless Speech-to-Speech Translation With Limited Parallel Data](https://mlsanity.com/paper/2305.15405/) (2023)
- [A multilingual training strategy for low resource Text to Speech](https://mlsanity.com/paper/2409.01217/) (2024)
- [Spatial HuBERT: Self-supervised Spatial Speech Representation Learning for a Single Talker from Multi-channel Audio](https://mlsanity.com/paper/2310.10922/) (2023)
- [Towards Unsupervised Speech Recognition at the Syllable-Level](https://mlsanity.com/paper/2510.03639/) (2025)

## Abstract (arXiv)

We introduce the first unsupervised speech synthesis system based on a simple, yet effective recipe. The framework leverages recent work in unsupervised speech recognition as well as existing neural-based speech synthesis. Using only unlabeled speech audio and unlabeled text as well as a lexicon, our method enables speech synthesis without the need for a human-labeled corpus. Experiments demonstrate the unsupervised system can synthesize speech similar to a supervised counterpart in terms of naturalness and intelligibility measured by human evaluation.