mlsanityData to 19 Oct 2026

Pre-training End-to-end ASR Models with Augmented Speech Samples Queried by Text

arXiv 2307.16332 ↗First posted 30 Jul 2023eess.AScited by 0

On the map: Speech and audio language models → Multilingual and low-resource ASR

Abstract (arXiv)

In end-to-end automatic speech recognition system, one of the difficulties for language expansion is the limited paired speech and text training data. In this paper, we propose a novel method to generate augmented samples with unpaired speech feature segments and text data for model pre-training, which has the advantage of low cost without using additional speech data. When mixing 20,000 hours augmented speech data generated by our method with 12,500 hours original transcribed speech data for Italian Transformer transducer model pre-training, we achieve 8.7% relative word error rate reduction. The pre-trained model achieves similar performance as the model pre-trained with multilingual transcribed 75,000 hours raw speech data. When merging the augmented speech data with the multilingual data to pre-train a new model, we achieve even more relative word error rate reduction of 12.2% over the baseline, which further verifies the effectiveness of our method for speech data augmentation.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Multilingual and low-resource ASR, one of 11 directions in Speech and audio language models. Builds on 26 of its 38 references that are in the map. First posted on arXiv 30 Jul 2023.

Builds on

26 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Semi-supervised transfer learning for language expansion of end-to-end speech recognition models to low-resource languages19 Nov
  2. 2021A Configurable Multilingual Model is All You Need to Recognize All Languages13 Jul
  3. 2021On-the-Fly Aligned Data Augmentation for Sequence-to-Sequence ASR3 Apr

Other references · 23

  1. 2017Attention Is All You Needcited by 61,180
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  4. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 286
  5. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 279
  6. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 254
  7. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 211
  8. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 187
  9. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 120
  10. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 117
  11. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 101
  12. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 87
11 more
  1. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 73
  2. 2018Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytescited by 61
  3. 2020Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systemscited by 51
  4. 2019Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entitiescited by 47
  5. 2018Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modelingcited by 36
  6. 2021Injecting Text in Self-Supervised Speech Pretrainingcited by 31
  7. 2019From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognitioncited by 23
  8. 2021On Addressing Practical Challenges for RNN-Transducercited by 21
  9. 2018Advancing Acoustic-to-Word CTC Modelcited by 19
  10. 2019Improving Performance of End-to-End ASR on Numeric Sequencescited by 18
  11. 2021UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Datasetcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translation
  2. 2024Zero Shot Text to Speech Augmentation for Automatic Speech Recognition on Low-Resource Accented Speech Corpora
  3. 2025An Exhaustive Evaluation of TTS- and VC-based Data Augmentation for ASR
  4. 2023Text Generation with Speech Synthesis for ASR Data Augmentation
  5. 2021Data Augmentation with Locally-time Reversed Speech for Automatic Speech Recognition
  6. 2023Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation
  7. 2023Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition
  8. 2021Data Augmentation Methods for End-to-end Speech Recognition on Distant-Talk Scenarios