# Weakly-supervised word-level pronunciation error detection in non-native English speech

arXiv: https://arxiv.org/abs/2106.03494 · HTML: https://mlsanity.com/paper/2106.03494/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://mlsanity.com/paper/1512.01274/) (2015)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014)
- [GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing](https://mlsanity.com/paper/1907.04433/) (2019)
- [ASR-Free Pronunciation Assessment](https://mlsanity.com/paper/2005.11902/) (2020)
- [Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention](https://mlsanity.com/paper/2012.14788/) (2020)
- [Towards Real-time Mispronunciation Detection in Kids' Speech](https://mlsanity.com/paper/2003.01765/) (2020)
- [Text-Conditioned Transformer for Automatic Pronunciation Error Detection](https://mlsanity.com/paper/2008.12424/) (2020)
- [Mispronunciation Detection in Non-native (L2) English with Uncertainty Modeling](https://mlsanity.com/paper/2101.06396/) (2021)

## Built on by (most cited first, 4 of 4)

- [AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation](https://mlsanity.com/paper/2507.12705/) (2025)
- [Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need](https://mlsanity.com/paper/2207.00774/) (2022)
- [Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information](https://mlsanity.com/paper/2203.01826/) (2022)
- [Dhvani: A Weakly-supervised Phonemic Error Detection and Personalized Feedback System for Hindi](https://mlsanity.com/paper/2506.02166/) (2025)

## Abstract (arXiv)

We propose a weakly-supervised model for word-level mispronunciation detection in non-native (L2) English speech. To train this model, phonetically transcribed L2 speech is not required and we only need to mark mispronounced words. The lack of phonetic transcriptions for L2 speech means that the model has to learn only from a weak signal of word-level mispronunciations. Because of that and due to the limited amount of mispronounced L2 speech, the model is more likely to overfit. To limit this risk, we train it in a multi-task setup. In the first task, we estimate the probabilities of word-level mispronunciation. For the second task, we use a phoneme recognizer trained on phonetically transcribed L1 speech that is easily accessible and can be automatically annotated. Compared to state-of-the-art approaches, we improve the accuracy of detecting word-level pronunciation errors in AUC metric by 30% on the GUT Isle Corpus of L2 Polish speakers, and by 21.5% on the Isle Corpus of L2 German and Italian speakers.