# Improving sequence-to-sequence speech recognition training with on-the-fly data augmentation

arXiv: https://arxiv.org/abs/1910.13296 · HTML: https://mlsanity.com/paper/1910.13296/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2019.

## Built on by (most cited first, 29 of 29)

- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?](https://mlsanity.com/paper/2106.01045/) (2021)
- [Instant One-Shot Word-Learning for Context-Specific Neural Sequence-to-Sequence Speech Recognition](https://mlsanity.com/paper/2107.02268/) (2021)
- [Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation](https://mlsanity.com/paper/2203.09866/) (2022)
- [Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videos](https://mlsanity.com/paper/2206.04523/) (2022)
- [Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation](https://mlsanity.com/paper/2104.11710/) (2021)
- [Recent Progress in the CUHK Dysarthric Speech Recognition System](https://mlsanity.com/paper/2201.05845/) (2022)
- [Multi-stage Large Language Model Correction for Speech Recognition](https://mlsanity.com/paper/2310.11532/) (2023)
- [Is "moby dick" a Whale or a Bird? Named Entities and Terminology in Speech Translation](https://mlsanity.com/paper/2109.07439/) (2021)
- [On-the-Fly Aligned Data Augmentation for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2104.01393/) (2021)
- [Adaptive multilingual speech recognition with pretrained models](https://mlsanity.com/paper/2205.12304/) (2022)
- [Language-agnostic Code-Switching in Sequence-To-Sequence Speech Recognition](https://mlsanity.com/paper/2210.08992/) (2022)
- [Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs](https://mlsanity.com/paper/2512.16378/) (2025)
- [Dealing with training and test segmentation mismatch: FBK@IWSLT2021](https://mlsanity.com/paper/2106.12607/) (2021)
- [Text-To-Speech Data Augmentation for Low Resource Speech Recognition](https://mlsanity.com/paper/2204.00291/) (2022)
- [Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translation](https://mlsanity.com/paper/2210.15398/) (2022)
- [SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations](https://mlsanity.com/paper/2212.09699/) (2022)
- [Unsupervised Document Embedding via Contrastive Augmentation](https://mlsanity.com/paper/2103.14542/) (2021)
- [Data Augmentation Policy Search for Long-Term Forecasting](https://mlsanity.com/paper/2405.00319/) (2024)
- [Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition](https://mlsanity.com/paper/2209.08326/) (2022)
- [Random Utterance Concatenation Based Data Augmentation for Improving Short-video Speech Recognition](https://mlsanity.com/paper/2210.15876/) (2022)
- [ImportantAug: a data augmentation agent for speech](https://mlsanity.com/paper/2112.07156/) (2021)
- [Improving Generalization of Deep Neural Network Acoustic Models with Length Perturbation and N-best Based Label Smoothing](https://mlsanity.com/paper/2203.15176/) (2022)
- [Multilingual Speech Translation with Unified Transformer: Huawei Noah's Ark Lab at IWSLT 2021](https://mlsanity.com/paper/2106.00197/) (2021)
- [SpiroMask: Measuring Lung Function Using Consumer-Grade Masks](https://mlsanity.com/paper/2201.09280/) (2022)
- [Handling Numeric Expressions in Automatic Speech Recognition](https://mlsanity.com/paper/2408.00004/) (2024)
- [Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language](https://mlsanity.com/paper/2502.00421/) (2025)
- [ChannelAugment: Improving generalization of multi-channel ASR by training with input channel randomization](https://mlsanity.com/paper/2109.11225/) (2021)
- [Graph Structure Based Data Augmentation Method](https://mlsanity.com/paper/2205.14619/) (2022)

## Abstract (arXiv)

Sequence-to-Sequence (S2S) models recently started to show state-of-the-art performance for automatic speech recognition (ASR). With these large and deep models overfitting remains the largest problem, outweighing performance improvements that can be obtained from better architectures. One solution to the overfitting problem is increasing the amount of available training data and the variety exhibited by the training data with the help of data augmentation. In this paper we examine the influence of three data augmentation methods on the performance of two S2S model architectures. One of the data augmentation method comes from literature, while two other methods are our own development - a time perturbation in the frequency domain and sub-sequence sampling. Our experiments on Switchboard and Fisher data show state-of-the-art performance for S2S models that are trained solely on the speech training data and do not use additional text data.