# Unify and Conquer: How Phonetic Feature Representation Affects Polyglot Text-To-Speech (TTS)

arXiv: https://arxiv.org/abs/2207.01547 · HTML: https://mlsanity.com/paper/2207.01547/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 4 Jul 2022.

## Builds on (most cited first, 21 of 21)

- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes](https://mlsanity.com/paper/1811.09021/) (2018)
- [Improving Variational Auto-Encoders using Householder Flow](https://mlsanity.com/paper/1611.09630/) (2016)
- [End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning](https://mlsanity.com/paper/1904.06508/) (2019)
- [Universal Neural Vocoding with Parallel WaveNet](https://mlsanity.com/paper/2102.01106/) (2021)
- [Phonological Features for 0-shot Multilingual Speech Synthesis](https://mlsanity.com/paper/2008.04107/) (2020)
- [Effect of data reduction on sequence-to-sequence neural TTS](https://mlsanity.com/paper/1811.06315/) (2018)
- [Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speech](https://mlsanity.com/paper/2106.12896/) (2021)
- [Unsupervised Polyglot Text To Speech](https://mlsanity.com/paper/1902.02263/) (2019)
- [Cross-lingual Low Resource Speaker Adaptation Using Phonological Features](https://mlsanity.com/paper/2111.09075/) (2021)
- [Generating Multilingual Voices Using Speaker Space Translation Based on Bilingual Speaker Data](https://mlsanity.com/paper/2004.04972/) (2020)
- [Cross-lingual Multispeaker Text-to-Speech under Limited-Data Scenario](https://mlsanity.com/paper/2005.10441/) (2020)
- [Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement](https://mlsanity.com/paper/2011.06392/) (2020)
- [Revisiting IPA-based Cross-lingual Text-to-speech](https://mlsanity.com/paper/2110.07187/) (2021)
- [Combining speakers of multiple languages to improve quality of neural voices](https://mlsanity.com/paper/2108.07737/) (2021)

## Built on by (most cited first, 5 of 5)

- [SQuId: Measuring Speech Naturalness in Many Languages](https://mlsanity.com/paper/2210.06324/) (2022)
- [PWESuite: Phonetic Word Embeddings and Tasks They Facilitate](https://mlsanity.com/paper/2304.02541/) (2023)
- [DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speech](https://mlsanity.com/paper/2306.14145/) (2023)
- [Modelling low-resource accents without accent-specific TTS frontend](https://mlsanity.com/paper/2301.04606/) (2023)
- [Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech](https://mlsanity.com/paper/2309.08255/) (2023)

## Abstract (arXiv)

An essential design decision for multilingual Neural Text-To-Speech (NTTS) systems is how to represent input linguistic features within the model. Looking at the wide variety of approaches in the literature, two main paradigms emerge, unified and separate representations. The former uses a shared set of phonetic tokens across languages, whereas the latter uses unique phonetic tokens for each language. In this paper, we conduct a comprehensive study comparing multilingual NTTS systems models trained with both representations. Our results reveal that the unified approach consistently achieves better cross-lingual synthesis with respect to both naturalness and accent. Separate representations tend to have an order of magnitude more tokens than unified ones, which may affect model capacity. For this reason, we carry out an ablation study to understand the interaction of the representation type with the size of the token embedding. We find that the difference between the two paradigms only emerges above a certain threshold embedding size. This study provides strong evidence that unified representations should be the preferred paradigm when building multilingual NTTS systems.