# Phonological Level wav2vec2-based Mispronunciation Detection and Diagnosis Method

arXiv: https://arxiv.org/abs/2311.07037 · HTML: https://mlsanity.com/paper/2311.07037/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 80 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [End-to-end Handwritten Paragraph Text Recognition Using a Vertical Attention Network](https://mlsanity.com/paper/2012.03868/) (2020)
- [Interpretable Convolutional Filters with SincNet](https://mlsanity.com/paper/1811.09725/) (2018)
- [A Full Text-Dependent End to End Mispronunciation Detection and Diagnosis with Easy Data Augmentation Techniques](https://mlsanity.com/paper/2104.08428/) (2021)
- [An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling](https://mlsanity.com/paper/2005.11950/) (2020)
- [An Effective End-to-End Modeling Approach for Mispronunciation Detection](https://mlsanity.com/paper/2005.08440/) (2020)
- [An Approach to Mispronunciation Detection and Diagnosis with Acoustic, Phonetic and Linguistic (APL) Embeddings](https://mlsanity.com/paper/2110.07274/) (2021)
- [Autosegmental Neural Nets: Should Phones and Tones be Synchronous or Asynchronous?](https://mlsanity.com/paper/2007.14351/) (2020)
- [Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods](https://mlsanity.com/paper/2108.11627/) (2021)
- [End-to-End Mispronunciation Detection and Diagnosis From Raw Waveforms](https://mlsanity.com/paper/2103.03023/) (2021)
- [Manner of Articulation Detection using Connectionist Temporal Classification to Improve Automatic Speech Recognition Performance](https://mlsanity.com/paper/1811.01644/) (2018)
- [Attention model for articulatory features detection](https://mlsanity.com/paper/1907.01914/) (2019)
- [Exploring Universal Speech Attributes for Speaker Verification with an Improved Cross-stitch Network](https://mlsanity.com/paper/2010.06248/) (2020)

## Built on by (most cited first, 3 of 3)

- [Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment](https://mlsanity.com/paper/2502.07029/) (2025)
- [Phoneme-Level Mispronunciation Screening in Polish-Speaking Children with an Explainable Assistant](https://mlsanity.com/paper/2606.25181/) (2026)
- [Multi-Task Learning for Non-Canonical Phoneme Recognition via Articulatory Feature Decomposition](https://mlsanity.com/paper/2608.22273/) (2026)

## Abstract (arXiv)

The automatic identification and analysis of pronunciation errors, known as Mispronunciation Detection and Diagnosis (MDD) plays a crucial role in Computer Aided Pronunciation Learning (CAPL) tools such as Second-Language (L2) learning or speech therapy applications. Existing MDD methods relying on analysing phonemes can only detect categorical errors of phonemes that have an adequate amount of training data to be modelled. With the unpredictable nature of the pronunciation errors of non-native or disordered speakers and the scarcity of training datasets, it is unfeasible to model all types of mispronunciations. Moreover, phoneme-level MDD approaches have a limited ability to provide detailed diagnostic information about the error made. In this paper, we propose a low-level MDD approach based on the detection of speech attribute features. Speech attribute features break down phoneme production into elementary components that are directly related to the articulatory system leading to more formative feedback to the learner. We further propose a multi-label variant of the Connectionist Temporal Classification (CTC) approach to jointly model the non-mutually exclusive speech attributes using a single model. The pre-trained wav2vec2 model was employed as a core model for the speech attribute detector. The proposed method was applied to L2 speech corpora collected from English learners from different native languages. The proposed speech attribute MDD method was further compared to the traditional phoneme-level MDD and achieved a significantly lower False Acceptance Rate (FAR), False Rejection Rate (FRR), and Diagnostic Error Rate (DER) over all speech attributes compared to the phoneme-level equivalent.