# Improving Large-Scale Weakly Supervised ASR by Filtering and Selection

arXiv: https://arxiv.org/abs/2606.28728 · HTML: https://mlsanity.com/paper/2606.28728/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [VoxLingua107: a Dataset for Spoken Language Recognition](https://mlsanity.com/paper/2011.12998/) (2020)
- [The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage](https://mlsanity.com/paper/2111.09344/) (2021)
- [OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer](https://mlsanity.com/paper/2401.16658/) (2024)
- [YODAS: Youtube-Oriented Dataset for Audio and Speech](https://mlsanity.com/paper/2406.00899/) (2024)
- [OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification](https://mlsanity.com/paper/2402.12654/) (2024)
- [Release of Pre-Trained Models for the Japanese Language](https://mlsanity.com/paper/2404.01657/) (2024)
- [Leveraging Large Text Corpora for End-to-End Speech Summarization](https://mlsanity.com/paper/2303.00978/) (2023)
- [Unsupervised Data Selection via Discrete Speech Representation for ASR](https://mlsanity.com/paper/2204.01981/) (2022)
- [DITTO: Data-efficient and Fair Targeted Subset Selection for ASR Accent Adaptation](https://mlsanity.com/paper/2110.04908/) (2021)

## Abstract (arXiv)

Leveraging large-scale weakly supervised datasets is crucial to train robust end-to-end automatic speech recognition (ASR) models. However, such datasets often contain noisy labels and lack domain specificity, limiting their effectiveness. To address these issues and make better use of weakly supervised datasets, we propose a novel training approach incorporating data filtering and selection. Our approach consists of three steps: pretraining on the entire dataset, continued pretraining on a filtered subset based on character error rate (CER), and fine-tuning on a small number of acoustically similar samples to the target domain, selected from the filtered subset. In experiments with a 90,000-hour weakly supervised Japanese dataset, the proposed filtering and selection methods synergistically reduced CER by up to 6.4% and 4.0%, respectively, even though these steps reused training samples already used in the first pretraining step.