# Continued Pretraining for Domain Adaptation of Wav2vec2.0 in Automatic Speech Recognition for Elementary Math Classroom Settings

arXiv: https://arxiv.org/abs/2405.13018 · HTML: https://mlsanity.com/paper/2405.13018/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 41 references that are in the map.
- First posted on arXiv 15 May 2024.

## Builds on (most cited first, 13 of 13)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts](https://mlsanity.com/paper/2211.11772/) (2022)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [End-to-End Monaural Multi-speaker ASR System without Pretraining](https://mlsanity.com/paper/1811.02062/) (2018)
- [Adaptation of Whisper models to child speech recognition](https://mlsanity.com/paper/2307.13008/) (2023)
- [Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens](https://mlsanity.com/paper/2402.02302/) (2024)
- [Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation](https://mlsanity.com/paper/2401.03468/) (2024)
- [Self-Supervised Adaptive AV Fusion Module for Pre-Trained ASR Models](https://mlsanity.com/paper/2312.13873/) (2023)

## Built on by (most cited first, 4 of 4)

- [Breaking the Transcription Bottleneck: Fine-tuning ASR Models for Extremely Low-Resource Fieldwork Languages](https://mlsanity.com/paper/2506.17459/) (2025)
- [Continued Pretraining for Low-Resource Swahili ASR: Achieving State-of-the-Art Performance with Minimal Labeled Data](https://mlsanity.com/paper/2603.11378/) (2026)
- [Examining Test-Time Adaptation for Personalized Child Speech Recognition](https://mlsanity.com/paper/2409.13095/) (2024)
- [Improving Speech Recognition of Named Entities in Classroom Speech with LLM Revision and Phonetic-Semantic Context](https://mlsanity.com/paper/2506.10779/) (2025)

## Abstract (arXiv)

Creating Automatic Speech Recognition (ASR) systems that are robust and resilient to classroom conditions is paramount to the development of AI tools to aid teachers and students. In this work, we study the efficacy of continued pretraining (CPT) in adapting Wav2vec2.0 to the classroom domain. We show that CPT is a powerful tool in that regard and reduces the Word Error Rate (WER) of Wav2vec2.0-based models by upwards of 10%. More specifically, CPT improves the model's robustness to different noises, microphones, classroom conditions as well as classroom demographics. Our CPT models show improved ability to generalize to different demographics unseen in the labeled finetuning data.