# A Full Text-Dependent End to End Mispronunciation Detection and Diagnosis with Easy Data Augmentation Techniques

arXiv: https://arxiv.org/abs/2104.08428 · HTML: https://mlsanity.com/paper/2104.08428/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 19 references that are in the map.
- First posted on arXiv 17 Apr 2021.

## Builds on (most cited first, 3 of 3)

- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Context-aware Goodness of Pronunciation for Computer-Assisted Pronunciation Training](https://mlsanity.com/paper/2008.08647/) (2020)
- [An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling](https://mlsanity.com/paper/2005.11950/) (2020)

## Built on by (most cited first, 19 of 19)

- [An Approach to Mispronunciation Detection and Diagnosis with Acoustic, Phonetic and Linguistic (APL) Embeddings](https://mlsanity.com/paper/2110.07274/) (2021)
- [Automatic Pronunciation Assessment -- A Review](https://mlsanity.com/paper/2310.13974/) (2023)
- [Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss](https://mlsanity.com/paper/2502.07575/) (2025)
- [Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need](https://mlsanity.com/paper/2207.00774/) (2022)
- [Conversational Process Modeling: Can Generative AI Empower Domain Experts in Creating and Redesigning Process Models?](https://mlsanity.com/paper/2304.11065/) (2023)
- [Text-Aware End-to-end Mispronunciation Detection and Diagnosis](https://mlsanity.com/paper/2206.07289/) (2022)
- [L1-aware Multilingual Mispronunciation Detection Framework](https://mlsanity.com/paper/2309.07719/) (2023)
- [Phonological Level wav2vec2-based Mispronunciation Detection and Diagnosis Method](https://mlsanity.com/paper/2311.07037/) (2023)
- [Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms](https://mlsanity.com/paper/2110.08731/) (2021)
- [SpeechBlender: Speech Augmentation Framework for Mispronunciation Data Generation](https://mlsanity.com/paper/2211.00923/) (2022)
- [Data-Driven Mispronunciation Pattern Discovery for Robust Speech Recognition](https://mlsanity.com/paper/2502.00583/) (2025)
- [Prompting Whisper for Improved Verbatim Transcription and End-to-end Miscue Detection](https://mlsanity.com/paper/2505.23627/) (2025)
- [Segmentation-free Goodness of Pronunciation](https://mlsanity.com/paper/2507.16838/) (2025)
- [Exploring Non-Autoregressive End-To-End Neural Modeling For English Mispronunciation Detection And Diagnosis](https://mlsanity.com/paper/2111.00844/) (2021)
- [CoCA-MDD: A Coupled Cross-Attention based Framework for Streaming Mispronunciation Detection and Diagnosis](https://mlsanity.com/paper/2111.08191/) (2021)
- [MuFFIN: Multifaceted Pronunciation Feedback Model with Interactive Hierarchical Neural Modeling](https://mlsanity.com/paper/2510.04956/) (2025)
- [Mispronunciation Detection and Diagnosis Without Model Training: A Retrieval-Based Approach](https://mlsanity.com/paper/2511.20107/) (2025)
- [Domain-Aware Mispronunciation Detection and Diagnosis Using Language-Specific Statistical Graphs](https://mlsanity.com/paper/2606.05569/) (2026)
- [Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal](https://mlsanity.com/paper/2606.19910/) (2026)

## Abstract (arXiv)

Recently, end-to-end mispronunciation detection and diagnosis (MD&D) systems has become a popular alternative to greatly simplify the model-building process of conventional hybrid DNN-HMM systems by representing complicated modules with a single deep network architecture. In this paper, in order to utilize the prior text in the end-to-end structure, we present a novel text-dependent model which is difference with sed-mdd, the model achieves a fully end-to-end system by aligning the audio with the phoneme sequences of the prior text inside the model through the attention mechanism. Moreover, the prior text as input will be a problem of imbalance between positive and negative samples in the phoneme sequence. To alleviate this problem, we propose three simple data augmentation methods, which effectively improve the ability of model to capture mispronounced phonemes. We conduct experiments on L2-ARCTIC, and our best performance improved from 49.29% to 56.08% in F-measure metric compared to the CNN-RNN-CTC model.