# L1-aware Multilingual Mispronunciation Detection Framework

arXiv: https://arxiv.org/abs/2309.07719 · HTML: https://mlsanity.com/paper/2309.07719/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 22 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 8 of 8)

- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment](https://mlsanity.com/paper/2205.03432/) (2022)
- [Context-aware Goodness of Pronunciation for Computer-Assisted Pronunciation Training](https://mlsanity.com/paper/2008.08647/) (2020)
- [3M: An Effective Multi-view, Multi-granularity, and Multi-aspect Modeling Approach to English Pronunciation Assessment](https://mlsanity.com/paper/2208.09110/) (2022)
- [A Full Text-Dependent End to End Mispronunciation Detection and Diagnosis with Easy Data Augmentation Techniques](https://mlsanity.com/paper/2104.08428/) (2021)
- [Multi-View Multi-Task Representation Learning for Mispronunciation Detection](https://mlsanity.com/paper/2306.01845/) (2023)
- [SpeechBlender: Speech Augmentation Framework for Mispronunciation Data Generation](https://mlsanity.com/paper/2211.00923/) (2022)

## Built on by (most cited first, 3 of 3)

- [Mispronunciation Detection and Diagnosis Without Model Training: A Retrieval-Based Approach](https://mlsanity.com/paper/2511.20107/) (2025)
- [Domain-Aware Mispronunciation Detection and Diagnosis Using Language-Specific Statistical Graphs](https://mlsanity.com/paper/2606.05569/) (2026)
- [Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal](https://mlsanity.com/paper/2606.19910/) (2026)

## Abstract (arXiv)

The phonological discrepancies between a speaker's native (L1) and the non-native language (L2) serves as a major factor for mispronunciation. This paper introduces a novel multilingual MDD architecture, L1-MultiMDD, enriched with L1-aware speech representation. An end-to-end speech encoder is trained on the input signal and its corresponding reference phoneme sequence. First, an attention mechanism is deployed to align the input audio with the reference phoneme sequence. Afterwards, the L1-L2-speech embedding are extracted from an auxiliary model, pretrained in a multi-task setup identifying L1 and L2 language, and are infused with the primary network. Finally, the L1-MultiMDD is then optimized for a unified multilingual phoneme recognition task using connectionist temporal classification (CTC) loss for the target languages: English, Arabic, and Mandarin. Our experiments demonstrate the effectiveness of the proposed L1-MultiMDD framework on both seen -- L2-ARTIC, LATIC, and AraVoiceL2v2; and unseen -- EpaDB and Speechocean762 datasets. The consistent gains in PER, and false rejection rate (FRR) across all target languages confirm our approach's robustness, efficacy, and generalizability.