# Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis

arXiv: https://arxiv.org/abs/2407.04047 · HTML: https://mlsanity.com/paper/2407.04047/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Builds on (most cited first, 19 of 19)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Speech Recognition with Augmented Synthesized Speech](https://mlsanity.com/paper/1909.11699/) (2019)
- [Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systems](https://mlsanity.com/paper/2011.11564/) (2020)
- [The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR](https://mlsanity.com/paper/2303.18110/) (2023)
- [SynthASR: Unlocking Synthetic Data for Speech Recognition](https://mlsanity.com/paper/2106.07803/) (2021)
- [Injecting Text in Self-Supervised Speech Pretraining](https://mlsanity.com/paper/2108.12226/) (2021)
- [REDAT: Accent-Invariant Representation for End-to-End ASR by Domain Adversarial Training with Relabeling](https://mlsanity.com/paper/2012.07353/) (2020)
- [Best of Both Worlds: Robust Accented Speech Recognition with Adversarial Transfer Learning](https://mlsanity.com/paper/2103.05834/) (2021)
- [Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognition](https://mlsanity.com/paper/2203.15796/) (2022)
- [Improving Accent Identification and Accented Speech Recognition Under a Framework of Self-supervised Learning](https://mlsanity.com/paper/2109.07349/) (2021)
- [AIPNet: Generative Adversarial Pre-training of Accent-invariant Networks for End-to-end Speech Recognition](https://mlsanity.com/paper/1911.11935/) (2019)
- [ASR data augmentation in low-resource settings using cross-lingual multi-speaker TTS and cross-lingual voice conversion](https://mlsanity.com/paper/2204.00618/) (2022)
- [Improving Accented Speech Recognition with Multi-Domain Training](https://mlsanity.com/paper/2303.07924/) (2023)
- [Accented Speech Recognition With Accent-specific Codebooks](https://mlsanity.com/paper/2310.15970/) (2023)
- [Multi-pass Training and Cross-information Fusion for Low-resource End-to-end Accented Speech Recognition](https://mlsanity.com/paper/2306.11309/) (2023)

## Built on by (most cited first, 5 of 5)

- [Quantifying Data Similarity Using Cross Learning](https://mlsanity.com/paper/2510.10866/) (2025)
- [How to Evaluate Speech Translation with Source-Aware Neural MT Metrics](https://mlsanity.com/paper/2511.03295/) (2025)
- [Automatic Speech Recognition for Non-Native English: Accuracy and Disfluency Handling](https://mlsanity.com/paper/2503.06924/) (2025)
- [Towards Pretraining Robust ASR Foundation Model with Acoustic-Aware Data Augmentation](https://mlsanity.com/paper/2505.20606/) (2025)
- [Contrastive Regularization for Accent-Robust ASR](https://mlsanity.com/paper/2605.03297/) (2026)

## Abstract (arXiv)

This paper investigates the use of unsupervised text-to-speech synthesis (TTS) as a data augmentation method to improve accented speech recognition. TTS systems are trained with a small amount of accented speech training data and their pseudo-labels rather than manual transcriptions, and hence unsupervised. This approach enables the use of accented speech data without manual transcriptions to perform data augmentation for accented speech recognition. Synthetic accented speech data, generated from text prompts by using the TTS systems, are then combined with available non-accented speech data to train automatic speech recognition (ASR) systems. ASR experiments are performed in a self-supervised learning framework using a Wav2vec2.0 model which was pre-trained on large amount of unsupervised accented speech data. The accented speech data for training the unsupervised TTS are read speech, selected from L2-ARCTIC and British Isles corpora, while spontaneous conversational speech from the Edinburgh international accents of English corpus are used as the evaluation data. Experimental results show that Wav2vec2.0 models which are fine-tuned to downstream ASR task with synthetic accented speech data, generated by the unsupervised TTS, yield up to 6.1% relative word error rate reductions compared to a Wav2vec2.0 baseline which is fine-tuned with the non-accented speech data from Librispeech corpus.