mlsanityData to 7 Oct 2026

Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systems

arXiv 2011.11564 ↗First posted 23 Nov 2020eess.AS cs.SDcited by 41

Not on the map. Why?

Abstract (arXiv)

Today, many state-of-the-art automatic speech recognition (ASR) systems apply all-neural models that map audio to word sequences trained end-to-end along one global optimisation criterion in a fully data driven fashion. These models allow high precision ASR for domains and words represented in the training material but have difficulties recognising words that are rarely or not at all represented during training, i.e. trending words and new named entities. In this paper, we use a text-to-speech (TTS) engine to provide synthetic audio for out-of-vocabulary (OOV) words. We aim to boost the recognition accuracy of a recurrent neural network transducer (RNN-T) on OOV words by using the extra audio-text pairs, while maintaining the performance on the non-OOV words. Different regularisation techniques are explored and the best performance is achieved by fine-tuning the RNN-T on both original training data and extra synthetic data with elastic weight consolidation (EWC) applied on the encoder. This yields a 57% relative word error rate (WER) reduction on utterances containing OOV words without any degradation on the whole test set.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 8; 2023: 14; 2024: 9; 2025: 4.

41 papers.

Sort:

By area:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  2. 2023Machine Learning for Synthetic Data Generation: A Reviewcited by 118
  3. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  4. 2021SynthASR: Unlocking Synthetic Data for Speech Recognitioncited by 34
  5. 2021Factorized Neural Transducer for Efficient Language Model Adaptationcited by 31
  6. 2021Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognitioncited by 24
  7. 2024Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognitioncited by 14
  8. 2022Improving the fusion of acoustic and text representations in RNN-Tcited by 11
  9. 2022Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition modelscited by 11
  10. 2023Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generatorcited by 11
  11. 2023JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognitioncited by 9
  12. 2024Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognitioncited by 9
29 more
  1. 2022Who Are We Talking About? Handling Person Names in Speech Translationcited by 8
  2. 2022A Simple Baseline for Domain Adaptation in End to End ASR Systems Using Synthetic Datacited by 7
  3. 2022When Is TTS Augmentation Through a Pivot Language Useful?cited by 7
  4. 2025A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Datacited by 7
  5. 2023OLISIA: a Cascade System for Spoken Dialogue State Trackingcited by 5
  6. 2023Retraining-free Customized ASR for Enharmonic Words Based on a Named-Entity-Aware Model and Phoneme Similarity Estimationcited by 5
  7. 2024Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesiscited by 5
  8. 2023On the Relevance of Phoneme Duration Variability of Synthesized Training Data for Automatic Speech Recognitioncited by 4
  9. 2022Random Utterance Concatenation Based Data Augmentation for Improving Short-video Speech Recognitioncited by 3
  10. 2022Stutter-TTS: Controlled Synthesis and Improved Recognition of Stuttered Speechcited by 3
  11. 2023Using Text Injection to Improve Recognition of Personal Identifiers in Speechcited by 3
  12. 2023Decoupled Structure for Improved Adaptability of End-to-End Modelscited by 3
  13. 2024Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approachcited by 3
  14. 2024Pre-training with Synthetic Patterns for Audiocited by 3
  15. 2023Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentationcited by 2
  16. 2023Improved Factorized Neural Transducer Model For text-only Domain Adaptationcited by 2
  17. 2021Data Incubation -- Synthesizing Missing Data for Handwriting Recognitioncited by 1
  18. 2023Improving Rare Words Recognition through Homophone Extension and Unified Writing for Low-resource Cantonese Speech Recognitioncited by 1
  19. 2023Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognitioncited by 1
  20. 2024Contrastive Learning from Synthetic Audio Doppelg"angerscited by 1
  21. 2024Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLMcited by 1
  22. 2025Improving Synthetic Data Training for Contextual Biasing Models with a Keyword-Aware Cost Functioncited by 1
  23. 2021Ensuring the Inclusive Use of Natural Language Processing in the Global Response to COVID-19cited by 0
  24. 2022Context-based out-of-vocabulary word recovery for ASR systems in Indian languagescited by 0
  25. 2023Incorporating Class-based Language Model for Named Entity Recognition in Factorized Neural Transducercited by 0
  26. 2024Initial Decoding with Minimally Augmented Language Model for Improved Lattice Rescoring in Low Resource ASRcited by 0
  27. 2024Evaluating Automatic Speech Recognition Systems for Korean Meteorological Expertscited by 0
  28. 2025Efficient Trie-based Biasing using K-step Prediction for Rare Word Recognitioncited by 0
  29. 2025WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformerscited by 0