# Building a mixed-lingual neural TTS system with only monolingual data

arXiv: https://arxiv.org/abs/1904.06063 · HTML: https://mlsanity.com/paper/1904.06063/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2019.

## Built on by (most cited first, 9 of 9)

- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [SwissDial: Parallel Multidialectal Corpus of Spoken Swiss German](https://mlsanity.com/paper/2103.11401/) (2021)
- [Revisiting IPA-based Cross-lingual Text-to-speech](https://mlsanity.com/paper/2110.07187/) (2021)
- [Combining speakers of multiple languages to improve quality of neural voices](https://mlsanity.com/paper/2108.07737/) (2021)
- [Improve Bilingual TTS Using Dynamic Language and Phonology Embedding](https://mlsanity.com/paper/2212.03435/) (2022)
- [Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classification](https://mlsanity.com/paper/2109.13486/) (2021)
- [Applying Phonological Features in Multilingual Text-To-Speech](https://mlsanity.com/paper/2110.03609/) (2021)
- [Improve Cross-lingual Voice Cloning Using Low-quality Code-switched Data](https://mlsanity.com/paper/2110.07210/) (2021)
- [Data-augmented cross-lingual synthesis in a teacher-student framework](https://mlsanity.com/paper/2204.00061/) (2022)

## Abstract (arXiv)

When deploying a Chinese neural text-to-speech (TTS) synthesis system, one of the challenges is to synthesize Chinese utterances with English phrases or words embedded. This paper looks into the problem in the encoder-decoder framework when only monolingual data from a target speaker is available. Specifically, we view the problem from two aspects: speaker consistency within an utterance and naturalness. We start the investigation with an Average Voice Model which is built from multi-speaker monolingual data, i.e. Mandarin and English data. On the basis of that, we look into speaker embedding for speaker consistency within an utterance and phoneme embedding for naturalness and intelligibility and study the choice of data for model training. We report the findings and discuss the challenges to build a mixed-lingual TTS system with only monolingual data.