# SynthASR: Unlocking Synthetic Data for Speech Recognition

arXiv: https://arxiv.org/abs/2106.07803 · HTML: https://mlsanity.com/paper/2106.07803/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition](https://mlsanity.com/paper/1912.01679/) (2019)
- [SpecAugment on Large Scale Datasets](https://mlsanity.com/paper/1912.05533/) (2019)
- [Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems](https://mlsanity.com/paper/1912.09257/) (2019)
- [Towards Fast and Accurate Streaming End-to-End ASR](https://mlsanity.com/paper/2004.11544/) (2020)
- [Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systems](https://mlsanity.com/paper/2011.11564/) (2020)
- [In Other News: A Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data](https://mlsanity.com/paper/1904.02790/) (2019)
- [Parallel WaveNet conditioned on VAE latent vectors](https://mlsanity.com/paper/2012.09703/) (2020)

## Built on by (most cited first, 30 of 34)

- [Machine Learning for Synthetic Data Generation: A Review](https://mlsanity.com/paper/2302.04062/) (2023)
- [On the Importance and Applicability of Pre-Training for Federated Learning](https://mlsanity.com/paper/2206.11488/) (2022)
- [SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision](https://mlsanity.com/paper/2303.17200/) (2023)
- [Building African Voices](https://mlsanity.com/paper/2207.00688/) (2022)
- [Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models](https://mlsanity.com/paper/2212.01992/) (2022)
- [Phoneme Hallucinator: One-shot Voice Conversion via Set Expansion](https://mlsanity.com/paper/2308.06382/) (2023)
- [Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition](https://mlsanity.com/paper/2408.09215/) (2024)
- [Continual Learning for Monolingual End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2112.09427/) (2021)
- [When Is TTS Augmentation Through a Pivot Language Useful?](https://mlsanity.com/paper/2207.09889/) (2022)
- [Real-Time Multimodal Cognitive Assistant for Emergency Medical Services](https://mlsanity.com/paper/2403.06734/) (2024)
- [USTED: Improving ASR with a Unified Speech and Text Encoder-Decoder](https://mlsanity.com/paper/2202.06045/) (2022)
- [A Simple Baseline for Domain Adaptation in End to End ASR Systems Using Synthetic Data](https://mlsanity.com/paper/2206.13240/) (2022)
- [Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLM](https://mlsanity.com/paper/2506.11089/) (2025)
- [Text Generation with Speech Synthesis for ASR Data Augmentation](https://mlsanity.com/paper/2305.16333/) (2023)
- [Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis](https://mlsanity.com/paper/2407.04047/) (2024)
- [Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need](https://mlsanity.com/paper/2207.00774/) (2022)
- [On-the-fly Text Retrieval for End-to-End ASR Adaptation](https://mlsanity.com/paper/2303.10942/) (2023)
- [Can Synthetic Audio From Generative Foundation Models Assist Audio Recognition and Speech Modeling?](https://mlsanity.com/paper/2406.08800/) (2024)
- [Using Text Injection to Improve Recognition of Personal Identifiers in Speech](https://mlsanity.com/paper/2308.07393/) (2023)
- [Hate Speech Detection in Limited Data Contexts using Synthetic Data Generation](https://mlsanity.com/paper/2310.02876/) (2023)
- [External Language Model Integration for Factorized Neural Transducers](https://mlsanity.com/paper/2305.17304/) (2023)
- [High-precision medical speech recognition through synthetic data and semantic correction: UNITED-MEDASR](https://mlsanity.com/paper/2412.00055/) (2024)
- [Contrastive Learning from Synthetic Audio Doppelg"angers](https://mlsanity.com/paper/2406.05923/) (2024)
- [Handling Numeric Expressions in Automatic Speech Recognition](https://mlsanity.com/paper/2408.00004/) (2024)
- [Towards Pretraining Robust ASR Foundation Model with Acoustic-Aware Data Augmentation](https://mlsanity.com/paper/2505.20606/) (2025)
- [Data Augmentation for Spoken Grammatical Error Correction](https://mlsanity.com/paper/2507.19374/) (2025)
- [Interpolating Speaker Identities in Embedding Space for Data Expansion](https://mlsanity.com/paper/2508.19210/) (2025)
- [LLM-to-Speech: A Synthetic Data Pipeline for Training Dialectal Text-to-Speech Models](https://mlsanity.com/paper/2602.15675/) (2026)
- [Improved Child Text-to-Speech Synthesis through Fastpitch-based Transfer Learning](https://mlsanity.com/paper/2311.04313/) (2023)
- [TI-ASU: Toward Robust Automatic Speech Understanding through Text-to-speech Imputation Against Missing Speech Modality](https://mlsanity.com/paper/2404.17983/) (2024)

## Abstract (arXiv)

End-to-end (E2E) automatic speech recognition (ASR) models have recently demonstrated superior performance over the traditional hybrid ASR models. Training an E2E ASR model requires a large amount of data which is not only expensive but may also raise dependency on production data. At the same time, synthetic speech generated by the state-of-the-art text-to-speech (TTS) engines has advanced to near-human naturalness. In this work, we propose to utilize synthetic speech for ASR training (SynthASR) in applications where data is sparse or hard to get for ASR model training. In addition, we apply continual learning with a novel multi-stage training strategy to address catastrophic forgetting, achieved by a mix of weighted multi-style training, data augmentation, encoder freezing, and parameter regularization. In our experiments conducted on in-house datasets for a new application of recognizing medication names, training ASR RNN-T models with synthetic audio via the proposed multi-stage training improved the recognition performance on new application by more than 65% relative, without degradation on existing general applications. Our observations show that SynthASR holds great promise in training the state-of-the-art large-scale E2E ASR models for new applications while reducing the costs and dependency on production data.