# Learning pronunciation from a foreign language in speech synthesis networks

arXiv: https://arxiv.org/abs/1811.09364 · HTML: https://mlsanity.com/paper/1811.09364/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2018.

## Built on by (most cited first, 12 of 12)

- [Unified Speech-Text Pre-training for Speech Translation and Recognition](https://mlsanity.com/paper/2204.05409/) (2022)
- [Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://mlsanity.com/paper/2107.05782/) (2021)
- [Pre-training for Speech Translation: CTC Meets Optimal Transport](https://mlsanity.com/paper/2301.11716/) (2023)
- [EURO: ESPnet Unsupervised ASR Open-source Toolkit](https://mlsanity.com/paper/2211.17196/) (2022)
- [Non-native English lexicon creation for bilingual speech synthesis](https://mlsanity.com/paper/2106.10870/) (2021)
- [A multilingual training strategy for low resource Text to Speech](https://mlsanity.com/paper/2409.01217/) (2024)
- [Towards Lifelong Learning of Multilingual Text-To-Speech Synthesis](https://mlsanity.com/paper/2110.04482/) (2021)
- [BiSinger: Bilingual Singing Voice Synthesis](https://mlsanity.com/paper/2309.14089/) (2023)
- [Improve Bilingual TTS Using Dynamic Language and Phonology Embedding](https://mlsanity.com/paper/2212.03435/) (2022)
- [RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modeling](https://mlsanity.com/paper/2505.22024/) (2025)
- [Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Data](https://mlsanity.com/paper/2506.19159/) (2025)
- [Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech](https://mlsanity.com/paper/2206.02147/) (2022)

## Abstract (arXiv)

Although there are more than 6,500 languages in the world, the pronunciations of many phonemes sound similar across the languages. When people learn a foreign language, their pronunciation often reflects their native language's characteristics. This motivates us to investigate how the speech synthesis network learns the pronunciation from datasets from different languages. In this study, we are interested in analyzing and taking advantage of multilingual speech synthesis network. First, we train the speech synthesis network bilingually in English and Korean and analyze how the network learns the relations of phoneme pronunciation between the languages. Our experimental result shows that the learned phoneme embedding vectors are located closer if their pronunciations are similar across the languages. Consequently, the trained networks can synthesize the English speakers' Korean speech and vice versa. Using this result, we propose a training framework to utilize information from a different language. To be specific, we pre-train a speech synthesis network using datasets from both high-resource language and low-resource language, then we fine-tune the network using the low-resource language dataset. Finally, we conducted more simulations on 10 different languages to show it is generally extendable to other languages.