# Accent conversion using discrete units with parallel data synthesized from controllable accented TTS

arXiv: https://arxiv.org/abs/2410.03734 · HTML: https://mlsanity.com/paper/2410.03734/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 13 of 13)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus](https://mlsanity.com/paper/2305.18802/) (2023)
- [Relative Positional Encoding for Speech Recognition and Direct Translation](https://mlsanity.com/paper/2005.09940/) (2020)
- [Voice-preserving Zero-shot Multiple Accent Conversion](https://mlsanity.com/paper/2211.13282/) (2022)
- [Efficient Weight factorization for Multilingual Speech Recognition](https://mlsanity.com/paper/2105.03010/) (2021)
- [Adaptive multilingual speech recognition with pretrained models](https://mlsanity.com/paper/2205.12304/) (2022)
- [Zero-Shot Accent Conversion using Pseudo Siamese Disentanglement Network](https://mlsanity.com/paper/2212.05751/) (2022)

## Built on by (most cited first, 5 of 5)

- [CosyAccent: Duration-Controllable Accent Normalization Using Source-Synthesis Training Data](https://mlsanity.com/paper/2602.19166/) (2026)
- [Streaming Non-Autoregressive Model for Accent Conversion and Pronunciation Improvement](https://mlsanity.com/paper/2506.16580/) (2025)
- [Controllable Accent Normalization via Discrete Diffusion](https://mlsanity.com/paper/2603.14275/) (2026)
- [Synthetic Audio Generation Framework for Air Traffic Control Speech Recognition](https://mlsanity.com/paper/2606.21340/) (2026)
- [TokAN: Accent Normalization Using Self-Supervised Speech Tokens](https://mlsanity.com/paper/2607.03928/) (2026)

## Abstract (arXiv)

The goal of accent conversion (AC) is to convert speech accents while preserving content and speaker identity. Previous methods either required reference utterances during inference, did not preserve speaker identity well, or used one-to-one systems that could only be trained for each non-native accent. This paper presents a promising AC model that can convert many accents into native to overcome these issues. Our approach utilizes discrete units, derived from clustering self-supervised representations of native speech, as an intermediary target for accent conversion. Leveraging multi-speaker text-to-speech synthesis, it transforms these discrete representations back into native speech while retaining the speaker identity. Additionally, we develop an efficient data augmentation method to train the system without demanding a lot of non-native resources. Our system is proved to improve non-native speaker fluency, sound like a native accent, and preserve original speaker identity well.