# Evaluating and reducing the distance between synthetic and real speech distributions

arXiv: https://arxiv.org/abs/2211.16049 · HTML: https://mlsanity.com/paper/2211.16049/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [High Fidelity Speech Synthesis with Adversarial Networks](https://mlsanity.com/paper/1909.11646/) (2019)
- [Speech Recognition with Augmented Synthesized Speech](https://mlsanity.com/paper/1909.11699/) (2019)
- [VoiceFixer: Toward General Speech Restoration with Neural Vocoder](https://mlsanity.com/paper/2109.13731/) (2021)
- [Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech](https://mlsanity.com/paper/2103.04088/) (2021)
- [LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search](https://mlsanity.com/paper/2102.04040/) (2021)
- [Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition](https://mlsanity.com/paper/2110.11479/) (2021)
- [Training Neural Speech Recognition Systems with Synthetic Speech Augmentation](https://mlsanity.com/paper/1811.00707/) (2018)
- [Speaker Generation](https://mlsanity.com/paper/2111.05095/) (2021)
- [TransFusion: Transcribing Speech with Multinomial Diffusion](https://mlsanity.com/paper/2210.07677/) (2022)

## Built on by (most cited first, 9 of 9)

- [TTSDS -- Text-to-Speech Distribution Score](https://mlsanity.com/paper/2407.12707/) (2024)
- [An Exhaustive Evaluation of TTS- and VC-based Data Augmentation for ASR](https://mlsanity.com/paper/2503.08954/) (2025)
- [On the Relevance of Phoneme Duration Variability of Synthesized Training Data for Automatic Speech Recognition](https://mlsanity.com/paper/2310.08132/) (2023)
- [Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech](https://mlsanity.com/paper/2310.00706/) (2023)
- [On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition](https://mlsanity.com/paper/2407.21476/) (2024)
- [Beyond Oversmoothing: Evaluating DDPM and MSE for Scalable Speech Synthesis in ASR](https://mlsanity.com/paper/2410.12279/) (2024)
- [Understanding Frechet Speech Distance for Synthetic Speech Quality Evaluation](https://mlsanity.com/paper/2601.21386/) (2026)
- [ZeSTA: Zero-Shot TTS Augmentation with Domain-Conditioned Training for Data-Efficient Personalized Speech Synthesis](https://mlsanity.com/paper/2603.04219/) (2026)
- [Error Rate is Not Enough: Costs of Augmentation in Low-Resource ASR](https://mlsanity.com/paper/2610.13807/) (2026)

## Abstract (arXiv)

While modern Text-to-Speech (TTS) systems can produce natural-sounding speech, they remain unable to reproduce the full diversity found in natural speech data. We consider the distribution of all possible real speech samples that could be generated by these speakers alongside the distribution of all synthetic samples that could be generated for the same set of speakers, using a particular TTS system. We set out to quantify the distance between real and synthetic speech via a range of utterance-level statistics related to properties of the speaker, speech prosody and acoustic environment. Differences in the distribution of these statistics are evaluated using the Wasserstein distance. We reduce these distances by providing ground-truth values at generation time, and quantify the improvements to the overall distribution distance, approximated using an automatic speech recognition system. Our best system achieves a 10% reduction in distribution distance.