# Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment

arXiv: https://arxiv.org/abs/2203.15937 · HTML: https://mlsanity.com/paper/2203.15937/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [Self-training and Pre-training are Complementary for Speech Recognition](https://mlsanity.com/paper/2010.11430/) (2020)
- [Exploring wav2vec 2.0 on speaker verification and language identification](https://mlsanity.com/paper/2012.06185/) (2020)
- [Iterative Pseudo-Labeling for Speech Recognition](https://mlsanity.com/paper/2005.09267/) (2020)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition](https://mlsanity.com/paper/2110.06309/) (2021)
- [Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition](https://mlsanity.com/paper/2106.08922/) (2021)
- [FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attention](https://mlsanity.com/paper/2010.14150/) (2020)
- [Semi-supervised ASR by End-to-end Self-training](https://mlsanity.com/paper/2001.09128/) (2020)
- [Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative](https://mlsanity.com/paper/2109.07437/) (2021)
- [An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling](https://mlsanity.com/paper/2005.11950/) (2020)
- [Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention](https://mlsanity.com/paper/2012.14788/) (2020)
- [An Effective End-to-End Modeling Approach for Mispronunciation Detection](https://mlsanity.com/paper/2005.08440/) (2020)
- [Leveraging native language information for improved accented speech recognition](https://mlsanity.com/paper/1904.09038/) (2019)

## Built on by (most cited first, 10 of 10)

- [What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model](https://mlsanity.com/paper/2306.06524/) (2023)
- [Automatic Pronunciation Assessment -- A Review](https://mlsanity.com/paper/2310.13974/) (2023)
- [Automatic Speech Recognition (ASR) for the Diagnosis of pronunciation of Speech Sound Disorders in Korean children](https://mlsanity.com/paper/2403.08187/) (2024)
- [An ensemble-based framework for mispronunciation detection of Arabic phonemes](https://mlsanity.com/paper/2301.01378/) (2023)
- [Unsupervised speech intelligibility assessment with utterance level alignment distance between teacher and learner Wav2Vec-2.0 representations](https://mlsanity.com/paper/2306.08845/) (2023)
- [Comparison of L2 Korean pronunciation error patterns from five L1 backgrounds by using automatic phonetic transcription](https://mlsanity.com/paper/2306.10821/) (2023)
- [The complementary roles of non-verbal cues for Robust Pronunciation Assessment](https://mlsanity.com/paper/2309.07739/) (2023)
- [English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM](https://mlsanity.com/paper/2509.02915/) (2025)
- [Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal](https://mlsanity.com/paper/2606.19910/) (2026)
- [Multi-Task Learning for Non-Canonical Phoneme Recognition via Articulatory Feature Decomposition](https://mlsanity.com/paper/2608.22273/) (2026)

## Abstract (arXiv)

Current leading mispronunciation detection and diagnosis (MDD) systems achieve promising performance via end-to-end phoneme recognition. One challenge of such end-to-end solutions is the scarcity of human-annotated phonemes on natural L2 speech. In this work, we leverage unlabeled L2 speech via a pseudo-labeling (PL) procedure and extend the fine-tuning approach based on pre-trained self-supervised learning (SSL) models. Specifically, we use Wav2vec 2.0 as our SSL model, and fine-tune it using original labeled L2 speech samples plus the created pseudo-labeled L2 speech samples. Our pseudo labels are dynamic and are produced by an ensemble of the online model on-the-fly, which ensures that our model is robust to pseudo label noise. We show that fine-tuning with pseudo labels achieves a 5.35% phoneme error rate reduction and 2.48% MDD F1 score improvement over a labeled-samples-only fine-tuning baseline. The proposed PL method is also shown to outperform conventional offline PL methods. Compared to the state-of-the-art MDD systems, our MDD solution produces a more accurate and consistent phonetic error diagnosis. In addition, we conduct an open test on a separate UTD-4Accents dataset, where our system recognition outputs show a strong correlation with human perception, based on accentedness and intelligibility.