# Text-Aware End-to-end Mispronunciation Detection and Diagnosis

arXiv: https://arxiv.org/abs/2206.07289 · HTML: https://mlsanity.com/paper/2206.07289/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 22 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 7 of 7)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [A Full Text-Dependent End to End Mispronunciation Detection and Diagnosis with Easy Data Augmentation Techniques](https://mlsanity.com/paper/2104.08428/) (2021)
- [An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling](https://mlsanity.com/paper/2005.11950/) (2020)
- [Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods](https://mlsanity.com/paper/2108.11627/) (2021)

## Built on by (most cited first, 3 of 3)

- [An ensemble-based framework for mispronunciation detection of Arabic phonemes](https://mlsanity.com/paper/2301.01378/) (2023)
- [Mispronunciation Detection and Diagnosis Without Model Training: A Retrieval-Based Approach](https://mlsanity.com/paper/2511.20107/) (2025)
- [Domain-Aware Mispronunciation Detection and Diagnosis Using Language-Specific Statistical Graphs](https://mlsanity.com/paper/2606.05569/) (2026)

## Abstract (arXiv)

Mispronunciation detection and diagnosis (MDD) technology is a key component of computer-assisted pronunciation training system (CAPT). In the field of assessing the pronunciation quality of constrained speech, the given transcriptions can play the role of a teacher. Conventional methods have fully utilized the prior texts for the model construction or improving the system performance, e.g. forced-alignment and extended recognition networks. Recently, some end-to-end based methods attempt to incorporate the prior texts into model training and preliminarily show the effectiveness. However, previous studies mostly consider applying raw attention mechanism to fuse audio representations with text representations, without taking possible text-pronunciation mismatch into account. In this paper, we present a gating strategy that assigns more importance to the relevant audio features while suppressing irrelevant text information. Moreover, given the transcriptions, we design an extra contrastive loss to reduce the gap between the learning objective of phoneme recognition and MDD. We conducted experiments using two publicly available datasets (TIMIT and L2-Arctic) and our best model improved the F1 score from 57.51% to 61.75% compared to the baselines. Besides, we provide a detailed analysis to shed light on the effectiveness of gating mechanism and contrastive learning on MDD.