# Towards Pretraining Robust ASR Foundation Model with Acoustic-Aware Data Augmentation

arXiv: https://arxiv.org/abs/2505.20606 · HTML: https://mlsanity.com/paper/2505.20606/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 23 of 23)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [Dialog Inpainting: Turning Documents into Dialogs](https://mlsanity.com/paper/2205.09073/) (2022)
- [SynthASR: Unlocking Synthetic Data for Speech Recognition](https://mlsanity.com/paper/2106.07803/) (2021)
- [iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform](https://mlsanity.com/paper/2203.02395/) (2022)
- [SpecMix : A Mixed Sample Data Augmentation method for Training withTime-Frequency Domain Features](https://mlsanity.com/paper/2108.03020/) (2021)
- [My Science Tutor (MyST) -- A Large Corpus of Children's Conversational Speech](https://mlsanity.com/paper/2309.13347/) (2023)
- [Training Data Augmentation for Dysarthric Automatic Speech Recognition by Text-to-Dysarthric-Speech Synthesis](https://mlsanity.com/paper/2406.08568/) (2024)
- [SpecAugment++: A Hidden Space Data Augmentation Method for Acoustic Scene Classification](https://mlsanity.com/paper/2103.16858/) (2021)
- [ASR data augmentation in low-resource settings using cross-lingual multi-speaker TTS and cross-lingual voice conversion](https://mlsanity.com/paper/2204.00618/) (2022)
- [SapAugment: Learning A Sample Adaptive Policy for Data Augmentation](https://mlsanity.com/paper/2011.01156/) (2020)
- [Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translation](https://mlsanity.com/paper/2210.15398/) (2022)
- [Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis](https://mlsanity.com/paper/2407.04047/) (2024)
- [On the Effect of Purely Synthetic Training Data for Different Automatic Speech Recognition Architectures](https://mlsanity.com/paper/2407.17997/) (2024)
- [FASA: a Flexible and Automatic Speech Aligner for Extracting High-quality Aligned Children Speech Data](https://mlsanity.com/paper/2406.17926/) (2024)
- [Automatic Screening for Children with Speech Disorder using Automatic Speech Recognition: Opportunities and Challenges](https://mlsanity.com/paper/2410.11865/) (2024)
- [G-Augment: Searching for the Meta-Structure of Data Augmentation Policies for ASR](https://mlsanity.com/paper/2210.10879/) (2022)
- [Sample adaptive data augmentation with progressive scheduling](https://mlsanity.com/paper/2412.00415/) (2024)

## Built on by (most cited first, 1 of 1)

- [NOPE-HYPE: A Structured Simulation Workflow for Robust Speech-to-Text Across Diverse Acoustic Environments](https://mlsanity.com/paper/2609.10058/) (2026)

## Abstract (arXiv)

Whisper's robust performance in automatic speech recognition (ASR) is often attributed to its massive 680k-hour training set, an impractical scale for most researchers. In this work, we examine how linguistic and acoustic diversity in training data affect the robustness of the ASR model and reveal that transcription generalization is primarily driven by acoustic variation rather than linguistic richness. We find that targeted acoustic augmentation methods could significantly improve the generalization ability of ASR models, reducing word-error rates by up to 19.24 percent on unseen datasets when training on the 960-hour Librispeech dataset. These findings highlight strategic acoustically focused data augmentation as a promising alternative to massive datasets for building robust ASR models, offering a potential solution to future foundation ASR models when massive human speech data is lacking.