# An Exhaustive Evaluation of TTS- and VC-based Data Augmentation for ASR

arXiv: https://arxiv.org/abs/2503.08954 · HTML: https://mlsanity.com/paper/2503.08954/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 66 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 30 of 42)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning](https://mlsanity.com/paper/2008.03648/) (2020)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model](https://mlsanity.com/paper/2104.05557/) (2021)
- [Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody prior](https://mlsanity.com/paper/2002.03788/) (2020)
- [Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition](https://mlsanity.com/paper/2110.11479/) (2021)
- [Improving Performance of End-to-End ASR on Numeric Sequences](https://mlsanity.com/paper/1907.01372/) (2019)
- [Improving Low Resource Code-switched ASR using Augmented Code-switched TTS](https://mlsanity.com/paper/2010.05549/) (2020)

## Built on by (most cited first, 5 of 5)

- [Enhancing ASR Performance in the Medical Domain for Dravidian Languages](https://mlsanity.com/paper/2604.19797/) (2026)
- [Adding Robust Code-Switching Capabilities to High Performance Multilingual ASR](https://mlsanity.com/paper/2606.21990/) (2026)
- [Synthetic Speech, Real Signal: Paralinguistic Preservation and Cross-Lingual Augmentation via Voice Cloning](https://mlsanity.com/paper/2607.22304/) (2026)
- [Phoneme-guided TTS augmentation for ASR: A unified pipeline and multilingual evaluation](https://mlsanity.com/paper/2608.26697/) (2026)
- [A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization](https://mlsanity.com/paper/2609.25471/) (2026)

## Abstract (arXiv)

Augmenting the training data of automatic speech recognition (ASR) systems with synthetic data generated by text-to-speech (TTS) or voice conversion (VC) has gained popularity in recent years. Several works have demonstrated improvements in ASR performance using this augmentation approach. However, because of the lower diversity of synthetic speech, naively combining synthetic and real data often does not yield the best results. In this work, we leverage recently proposed flow-based TTS/VC models allowing greater speech diversity, and assess the respective impact of augmenting various speech attributes on the word error rate (WER) achieved by several ASR models. Pitch augmentation and VC-based speaker augmentation are found to be ineffective in our setup. Jointly augmenting all other attributes reduces the WER of a Conformer-Transducer model by 11% relative on Common Voice and by up to 35% relative on LibriSpeech compared to training on real data only.