# Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need arXiv: https://arxiv.org/abs/2207.00774 ยท HTML: https://mlsanity.com/paper/2207.00774/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 85 references that are in the map. - First posted on arXiv 2 Jul 2022. ## Builds on (most cited first, 22 of 22) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014) - [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019) - [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015) - [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017) - [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015) - [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014) - [SynthASR: Unlocking Synthetic Data for Speech Recognition](https://mlsanity.com/paper/2106.07803/) (2021) - [Low-resource expressive text-to-speech using data augmentation](https://mlsanity.com/paper/2011.05707/) (2020) - [Universal Neural Vocoding with Parallel WaveNet](https://mlsanity.com/paper/2102.01106/) (2021) - [A Full Text-Dependent End to End Mispronunciation Detection and Diagnosis with Easy Data Augmentation Techniques](https://mlsanity.com/paper/2104.08428/) (2021) - [An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling](https://mlsanity.com/paper/2005.11950/) (2020) - [Effect of data reduction on sequence-to-sequence neural TTS](https://mlsanity.com/paper/1811.06315/) (2018) - [Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speech](https://mlsanity.com/paper/2106.12896/) (2021) - [ASR-Free Pronunciation Assessment](https://mlsanity.com/paper/2005.11902/) (2020) - [Text-free non-parallel many-to-many voice conversion using normalising flows](https://mlsanity.com/paper/2203.08009/) (2022) - [Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention](https://mlsanity.com/paper/2012.14788/) (2020) - [Towards Real-time Mispronunciation Detection in Kids' Speech](https://mlsanity.com/paper/2003.01765/) (2020) - [Text-Conditioned Transformer for Automatic Pronunciation Error Detection](https://mlsanity.com/paper/2008.12424/) (2020) - [Mispronunciation Detection in Non-native (L2) English with Uncertainty Modeling](https://mlsanity.com/paper/2101.06396/) (2021) - [Weakly-supervised word-level pronunciation error detection in non-native English speech](https://mlsanity.com/paper/2106.03494/) (2021) - [Maximum F1-score training for end-to-end mispronunciation detection and diagnosis of L2 English speech](https://mlsanity.com/paper/2108.13816/) (2021) ## Built on by (most cited first, 4 of 4) - [Preserving Phonemic Distinctions for Ordinal Regression: A Novel Loss Function for Automatic Pronunciation Assessment](https://mlsanity.com/paper/2310.01839/) (2023) - [Exploring the Viability of Synthetic Audio Data for Audio-Based Dialogue State Tracking](https://mlsanity.com/paper/2312.01842/) (2023) - [Dhvani: A Weakly-supervised Phonemic Error Detection and Personalized Feedback System for Hindi](https://mlsanity.com/paper/2506.02166/) (2025) - [MuFFIN: Multifaceted Pronunciation Feedback Model with Interactive Hierarchical Neural Modeling](https://mlsanity.com/paper/2510.04956/) (2025) ## Abstract (arXiv) The research community has long studied computer-assisted pronunciation training (CAPT) methods in non-native speech. Researchers focused on studying various model architectures, such as Bayesian networks and deep learning methods, as well as on the analysis of different representations of the speech signal. Despite significant progress in recent years, existing CAPT methods are not able to detect pronunciation errors with high accuracy (only 60% precision at 40%-80% recall). One of the key problems is the low availability of mispronounced speech that is needed for the reliable training of pronunciation error detection models. If we had a generative model that could mimic non-native speech and produce any amount of training data, then the task of detecting pronunciation errors would be much easier. We present three innovative techniques based on phoneme-to-phoneme (P2P), text-to-speech (T2S), and speech-to-speech (S2S) conversion to generate correctly pronounced and mispronounced synthetic speech. We show that these techniques not only improve the accuracy of three machine learning models for detecting pronunciation errors but also help establish a new state-of-the-art in the field. Earlier studies have used simple speech generation techniques such as P2P conversion, but only as an additional mechanism to improve the accuracy of pronunciation error detection. We, on the other hand, consider speech generation to be the first-class method of detecting pronunciation errors. The effectiveness of these techniques is assessed in the tasks of detecting pronunciation and lexical stress errors. Non-native English speech corpora of German, Italian, and Polish speakers are used in the evaluations. The best proposed S2S technique improves the accuracy of detecting pronunciation errors in AUC metric by 41% from 0.528 to 0.749 compared to the state-of-the-art approach.