# From Weak Labels to Strong Results: Utilizing 5,000 Hours of Noisy Classroom Transcripts with Minimal Accurate Data

arXiv: https://arxiv.org/abs/2505.17088 · HTML: https://mlsanity.com/paper/2505.17088/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 13 of 13)

- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Self-Training: A Survey](https://mlsanity.com/paper/2202.12040/) (2022)
- [The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts](https://mlsanity.com/paper/2211.11772/) (2022)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Models](https://mlsanity.com/paper/2406.10507/) (2024)
- [Kid-Whisper: Towards Bridging the Performance Gap in Automatic Speech Recognition for Children VS. Adults](https://mlsanity.com/paper/2309.07927/) (2023)
- [End-to-End Monaural Multi-speaker ASR System without Pretraining](https://mlsanity.com/paper/1811.02062/) (2018)
- [CPT-Boosted Wav2vec2.0: Towards Noise Robust Speech Recognition for Classroom Environments](https://mlsanity.com/paper/2409.14494/) (2024)
- [Self-Supervised Adaptive AV Fusion Module for Pre-Trained ASR Models](https://mlsanity.com/paper/2312.13873/) (2023)

## Abstract (arXiv)

Recent progress in speech recognition has relied on models trained on vast amounts of labeled data. However, classroom Automatic Speech Recognition (ASR) faces the real-world challenge of abundant weak transcripts paired with only a small amount of accurate, gold-standard data. In such low-resource settings, high transcription costs make re-transcription impractical. To address this, we ask: what is the best approach when abundant inexpensive weak transcripts coexist with limited gold-standard data, as is the case for classroom speech data? We propose Weakly Supervised Pretraining (WSP), a two-step process where models are first pretrained on weak transcripts in a supervised manner, and then fine-tuned on accurate data. Our results, based on both synthetic and real weak transcripts, show that WSP outperforms alternative methods, establishing it as an effective training methodology for low-resource ASR in real-world scenarios.