# On the Effect of Purely Synthetic Training Data for Different Automatic Speech Recognition Architectures

arXiv: https://arxiv.org/abs/2407.17997 · HTML: https://mlsanity.com/paper/2407.17997/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 25 Jul 2024.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Seed-TTS: A Family of High-Quality Versatile Speech Generation Models](https://mlsanity.com/paper/2406.02430/) (2024)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation](https://mlsanity.com/paper/2005.07157/) (2020)
- [Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition](https://mlsanity.com/paper/2110.11479/) (2021)
- [EAT: Enhanced ASR-TTS for Self-supervised Speech Recognition](https://mlsanity.com/paper/2104.07474/) (2021)
- [End-to-End Feedback Loss in Speech Chain Framework via Straight-Through Estimator](https://mlsanity.com/paper/1810.13107/) (2018)
- [Comparing the Benefit of Synthetic Training Data for Various Automatic Speech Recognition Architectures](https://mlsanity.com/paper/2104.05379/) (2021)
- [VRAIN-UPV MLLP's system for the Blizzard Challenge 2021](https://mlsanity.com/paper/2110.15792/) (2021)
- [On the Relevance of Phoneme Duration Variability of Synthesized Training Data for Automatic Speech Recognition](https://mlsanity.com/paper/2310.08132/) (2023)

## Built on by (most cited first, 4 of 4)

- [Towards Pretraining Robust ASR Foundation Model with Acoustic-Aware Data Augmentation](https://mlsanity.com/paper/2505.20606/) (2025)
- [Fairness in Dysarthric Speech Synthesis: Understanding Intrinsic Bias in Dysarthric Speech Cloning using F5-TTS](https://mlsanity.com/paper/2508.05102/) (2025)
- [Towards Real-world Environment-aware Zero-shot Text-to-speech Synthesis via Disentangled Audio Infilling](https://mlsanity.com/paper/2608.03011/) (2026)
- [A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization](https://mlsanity.com/paper/2609.25471/) (2026)

## Abstract (arXiv)

In this work we evaluate the utility of synthetic data for training automatic speech recognition (ASR). We use the ASR training data to train a text-to-speech (TTS) system similar to FastSpeech-2. With this TTS we reproduce the original training data, training ASR systems solely on synthetic data. For ASR, we use three different architectures, attention-based encoder-decoder, hybrid deep neural network hidden Markov model and a Gaussian mixture hidden Markov model, showing the different sensitivity of the models to synthetic data generation. In order to extend previous work, we present a number of ablation studies on the effectiveness of synthetic vs. real training data for ASR. In particular we focus on how the gap between training on synthetic and real data changes by varying the speaker embedding or by scaling the model size. For the latter we show that the TTS models generalize well, even when training scores indicate overfitting.