# Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information

arXiv: https://arxiv.org/abs/2203.01826 · HTML: https://mlsanity.com/paper/2203.01826/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 1 Mar 2022.

## Builds on (most cited first, 9 of 9)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment](https://mlsanity.com/paper/2104.01378/) (2021)
- [Deep-FSMN for Large Vocabulary Continuous Speech Recognition](https://mlsanity.com/paper/1803.05030/) (2018)
- [Context-aware Goodness of Pronunciation for Computer-Assisted Pronunciation Training](https://mlsanity.com/paper/2008.08647/) (2020)
- [An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling](https://mlsanity.com/paper/2005.11950/) (2020)
- [Multilingual Speech Evaluation: Case Studies on English, Malay and Tamil](https://mlsanity.com/paper/2107.03675/) (2021)
- [Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention](https://mlsanity.com/paper/2012.14788/) (2020)
- [Weakly-supervised word-level pronunciation error detection in non-native English speech](https://mlsanity.com/paper/2106.03494/) (2021)

## Built on by (most cited first, 3 of 3)

- [Automatic Pronunciation Assessment -- A Review](https://mlsanity.com/paper/2310.13974/) (2023)
- [SpeechBlender: Speech Augmentation Framework for Mispronunciation Data Generation](https://mlsanity.com/paper/2211.00923/) (2022)
- [Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal](https://mlsanity.com/paper/2606.19910/) (2026)

## Abstract (arXiv)

Deep learning-based pronunciation scoring models highly rely on the availability of the annotated non-native data, which is costly and has scalability issues. To deal with the data scarcity problem, data augmentation is commonly used for model pretraining. In this paper, we propose a phone-level mixup, a simple yet effective data augmentation method, to improve the performance of word-level pronunciation scoring. Specifically, given a phoneme sequence from lexicon, the artificial augmented word sample can be generated by randomly sampling from the corresponding phone-level features in training data, while the word score is the average of their GOP scores. Benefit from the arbitrary phone-level combination, the mixup is able to generate any word with various pronunciation scores. Moreover, we utilize multi-source information (e.g., MFCC and deep features) to further improve the scoring system performance. The experiments conducted on the Speechocean762 show that the proposed system outperforms the baseline by adding the mixup data for pretraining, with Pearson correlation coefficients (PCC) increasing from 0.567 to 0.61. The results also indicate that proposed method achieves similar performance by using 1/10 unlabeled data of baseline. In addition, the experimental results also demonstrate the efficiency of our proposed multi-source approach.