mlsanityData to 7 Oct 2026

Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition

arXiv 2110.11479 ↗First posted 21 Oct 2021eess.AS cs.LG cs.SDcited by 24

Not on the map. Why?

Abstract (arXiv)

With recent advances in speech synthesis, synthetic data is becoming a viable alternative to real data for training speech recognition models. However, machine learning with synthetic data is not trivial due to the gap between the synthetic and the real data distributions. Synthetic datasets may contain artifacts that do not exist in real data such as structured noise, content errors, or unrealistic speaking styles. Moreover, the synthesis process may introduce a bias due to uneven sampling of the data manifold. We propose two novel techniques during training to mitigate the problems due to the distribution gap: (i) a rejection sampling algorithm and (ii) using separate batch normalization statistics for the real and the synthetic samples. We show that these methods significantly improve the training of speech recognition models using synthetic data. We evaluate the proposed approach on keyword detection and Automatic Speech Recognition (ASR) tasks, and observe up to 18% and 13% relative error reduction, respectively, compared to naively using the synthetic data.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

27 of 38 references.

References · 27

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  4. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  5. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  8. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  9. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 287
  10. 2019Adversarial Examples Improve Image Recognitioncited by 283
  11. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  12. 2017Data Augmentation Generative Adversarial Networkscited by 278
15 more
  1. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  2. 2019Speech Recognition with Augmented Synthesized Speechcited by 60
  3. 2018Discriminator Rejection Samplingcited by 48
  4. 2019Generating Synthetic Audio Data for Attention-Based Speech Recognition Systemscited by 46
  5. 2019MelNet: A Generative Model for Audio in the Frequency Domaincited by 41
  6. 2020Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systemscited by 41
  7. 2018Cycle-consistency training for end-to-end speech recognitioncited by 31
  8. 2018Back-Translation-Style Data Augmentation for End-to-End ASRcited by 28
  9. 2020Training Keyword Spotters with Limited and Synthesized Speech Datacited by 20
  10. 2018Variational Rejection Samplingcited by 13
  11. 2021EAT: Enhanced ASR-TTS for Self-supervised Speech Recognitioncited by 13
  12. 2018End-to-End Feedback Loss in Speech Chain Framework via Straight-Through Estimatorcited by 11
  13. 2020Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesiscited by 11
  14. 2021Style Equalization: Unsupervised Learning of Controllable Generative Sequence Modelscited by 11
  15. 2020Optimize what matters: Training DNN-HMM Keyword Spotting Model Using End Metriccited by 9

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 8; 2024: 10; 2025: 4.

24 papers.

Sort:

By area:

  1. 2023Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generatorcited by 11
  2. 2024Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gapcited by 11
  3. 2023Text is All You Need: Personalizing ASR Models using Controllable Speech Synthesiscited by 9
  4. 2023Phoneme Hallucinator: One-shot Voice Conversion via Set Expansioncited by 9
  5. 2024Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognitioncited by 9
  6. 2022Evaluating and reducing the distance between synthetic and real speech distributionscited by 8
  7. 2025GAITGen: Disentangled Motion-Pathology Impaired Gait Generative Model -- Bringing Motion Generation to the Clinical Domaincited by 7
  8. 2023Text Generation with Speech Synthesis for ASR Data Augmentationcited by 5
  9. 2023Towards Selection of Text-to-speech Data to Augment ASR Trainingcited by 5
  10. 2024Revisiting ASR Error Correction with Specialized Modelscited by 5
  11. 2025An Exhaustive Evaluation of TTS- and VC-based Data Augmentation for ASRcited by 5
  12. 2023On the Relevance of Phoneme Duration Variability of Synthesized Training Data for Automatic Speech Recognitioncited by 4
12 more
  1. 2024On the Effect of Purely Synthetic Training Data for Different Automatic Speech Recognition Architecturescited by 4
  2. 2024Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuningcited by 4
  3. 2023Hate Speech Detection in Limited Data Contexts using Synthetic Data Generationcited by 3
  4. 2024On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognitioncited by 3
  5. 2025GReaTER: Generate Realistic Tabular data after data Enhancement and Reductioncited by 3
  6. 2025Synthetic Voice Data for Automatic Speech Recognition in African Languagescited by 3
  7. 2023Exploring the Viability of Synthetic Audio Data for Audio-Based Dialogue State Trackingcited by 1
  8. 2024Contrastive Learning from Synthetic Audio Doppelg"angerscited by 1
  9. 2024Adversarial training of Keyword Spotting to Minimize TTS Data Overfittingcited by 1
  10. 2024Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLMcited by 1
  11. 2022Crowdsourced-based Deep Convolutional Networks for Urban Flood Depth Mappingcited by 0
  12. 2024Beyond Oversmoothing: Evaluating DDPM and MSE for Scalable Speech Synthesis in ASRcited by 0