# Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2203.15796 · HTML: https://mlsanity.com/paper/2203.15796/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://mlsanity.com/paper/1710.07654/) (2017)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://mlsanity.com/paper/1710.08969/) (2017)
- [CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages](https://mlsanity.com/paper/1903.11269/) (2019)
- [Text-Free Image-to-Speech Synthesis Using Learned Segmental Units](https://mlsanity.com/paper/2012.15454/) (2020)
- [LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition](https://mlsanity.com/paper/2008.03687/) (2020)
- [Almost Unsupervised Text to Speech and Automatic Speech Recognition](https://mlsanity.com/paper/1905.06791/) (2019)
- [Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matching](https://mlsanity.com/paper/1812.09323/) (2018)
- [Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning](https://mlsanity.com/paper/1910.12729/) (2019)
- [Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languages](https://mlsanity.com/paper/2008.04549/) (2020)
- [Show and Speak: Directly Synthesize Spoken Description of Images](https://mlsanity.com/paper/2010.12267/) (2020)

## Built on by (most cited first, 14 of 14)

- [Simple and Effective Unsupervised Speech Translation](https://mlsanity.com/paper/2210.10191/) (2022)
- [Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining](https://mlsanity.com/paper/2301.12596/) (2023)
- [Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2407.13782/) (2024)
- [Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE](https://mlsanity.com/paper/2206.02512/) (2022)
- [Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speech](https://mlsanity.com/paper/2303.11131/) (2023)
- [Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation](https://mlsanity.com/paper/2305.07455/) (2023)
- [Unsupervised Speech Recognition with N-Skipgram and Positional Unigram Matching](https://mlsanity.com/paper/2310.02382/) (2023)
- [Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis](https://mlsanity.com/paper/2407.04047/) (2024)
- [A Theory of Unsupervised Speech Recognition](https://mlsanity.com/paper/2306.07926/) (2023)
- [A multilingual training strategy for low resource Text to Speech](https://mlsanity.com/paper/2409.01217/) (2024)
- [Iteratively Improving Speech Recognition and Voice Conversion](https://mlsanity.com/paper/2305.15055/) (2023)
- [Towards Unsupervised Speech Recognition at the Syllable-Level](https://mlsanity.com/paper/2510.03639/) (2025)
- [Towards Unsupervised Speech Recognition Without Pronunciation Models](https://mlsanity.com/paper/2406.08380/) (2024)
- [Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initialization](https://mlsanity.com/paper/2402.01692/) (2024)

## Abstract (arXiv)

An unsupervised text-to-speech synthesis (TTS) system learns to generate speech waveforms corresponding to any written sentence in a language by observing: 1) a collection of untranscribed speech waveforms in that language; 2) a collection of texts written in that language without access to any transcribed speech. Developing such a system can significantly improve the availability of speech technology to languages without a large amount of parallel speech and text data. This paper proposes an unsupervised TTS system based on an alignment module that outputs pseudo-text and another synthesis module that uses pseudo-text for training and real text for inference. Our unsupervised system can achieve comparable performance to the supervised system in seven languages with about 10-20 hours of speech each. A careful study on the effect of text units and vocoders has also been conducted to better understand what factors may affect unsupervised TTS performance. The samples generated by our models can be found at https://cactuswiththoughts.github.io/UnsupTTS-Demo, and our code can be found at https://github.com/lwang114/UnsupTTS.