# Generating Multilingual Voices Using Speaker Space Translation Based on Bilingual Speaker Data

arXiv: https://arxiv.org/abs/2004.04972 · HTML: https://mlsanity.com/paper/2004.04972/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2020.

## Built on by (most cited first, 7 of 7)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Talking Face Generation with Multilingual TTS](https://mlsanity.com/paper/2205.06421/) (2022)
- [Unify and Conquer: How Phonetic Feature Representation Affects Polyglot Text-To-Speech (TTS)](https://mlsanity.com/paper/2207.01547/) (2022)
- [Combining speakers of multiple languages to improve quality of neural voices](https://mlsanity.com/paper/2108.07737/) (2021)
- [LASPA: Language Agnostic Speaker Disentanglement with Prefix-Tuned Cross-Attention](https://mlsanity.com/paper/2506.02083/) (2025)
- [Improve Bilingual TTS Using Dynamic Language and Phonology Embedding](https://mlsanity.com/paper/2212.03435/) (2022)
- [G4G:A Generic Framework for High Fidelity Talking Face Generation with Fine-grained Intra-modal Alignment](https://mlsanity.com/paper/2402.18122/) (2024)

## Abstract (arXiv)

We present progress towards bilingual Text-to-Speech which is able to transform a monolingual voice to speak a second language while preserving speaker voice quality. We demonstrate that a bilingual speaker embedding space contains a separate distribution for each language and that a simple transform in speaker space generated by the speaker embedding can be used to control the degree of accent of a synthetic voice in a language. The same transform can be applied even to monolingual speakers. In our experiments speaker data from an English-Spanish (Mexican) bilingual speaker was used, and the goal was to enable English speakers to speak Spanish and Spanish speakers to speak English. We found that the simple transform was sufficient to convert a voice from one language to the other with a high degree of naturalness. In one case the transformed voice outperformed a native language voice in listening tests. Experiments further indicated that the transform preserved many of the characteristics of the original voice. The degree of accent present can be controlled and naturalness is relatively consistent across a range of accent values.