mlsanityData to 7 Oct 2026

Unify and Conquer: How Phonetic Feature Representation Affects Polyglot Text-To-Speech (TTS)

arXiv 2207.01547 ↗First posted 4 Jul 2022eess.AS cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

An essential design decision for multilingual Neural Text-To-Speech (NTTS) systems is how to represent input linguistic features within the model. Looking at the wide variety of approaches in the literature, two main paradigms emerge, unified and separate representations. The former uses a shared set of phonetic tokens across languages, whereas the latter uses unique phonetic tokens for each language. In this paper, we conduct a comprehensive study comparing multilingual NTTS systems models trained with both representations. Our results reveal that the unified approach consistently achieves better cross-lingual synthesis with respect to both naturalness and accent. Separate representations tend to have an order of magnitude more tokens than unified ones, which may affect model capacity. For this reason, we carry out an ablation study to understand the interaction of the representation type with the size of the token embedding. We find that the difference between the two paradigms only emerges above a certain threshold embedding size. This study provides strong evidence that unified representations should be the preferred paradigm when building multilingual NTTS systems.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 4 Jul 2022.

Builds on

21 of 31 references.

References · 21

  1. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  2. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  3. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  4. 2015Attention-Based Models for Speech Recognitioncited by 400
  5. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  6. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  7. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
  8. 2018Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytescited by 54
  9. 2016Improving Variational Auto-Encoders using Householder Flowcited by 41
  10. 2019End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learningcited by 30
  11. 2021Universal Neural Vocoding with Parallel WaveNetcited by 25
  12. 2020Phonological Features for 0-shot Multilingual Speech Synthesiscited by 20
9 more
  1. 2018Effect of data reduction on sequence-to-sequence neural TTScited by 16
  2. 2021Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speechcited by 14
  3. 2019Unsupervised Polyglot Text To Speechcited by 10
  4. 2021Cross-lingual Low Resource Speaker Adaptation Using Phonological Featurescited by 8
  5. 2020Generating Multilingual Voices Using Speaker Space Translation Based on Bilingual Speaker Datacited by 7
  6. 2020Cross-lingual Multispeaker Text-to-Speech under Limited-Data Scenariocited by 6
  7. 2020Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancementcited by 6
  8. 2021Revisiting IPA-based Cross-lingual Text-to-speechcited by 4
  9. 2021Combining speakers of multiple languages to improve quality of neural voicescited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4.

5 papers.

  1. 2022SQuId: Measuring Speech Naturalness in Many Languagescited by 10
  2. 2023PWESuite: Phonetic Word Embeddings and Tasks They Facilitatecited by 6
  3. 2023DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speechcited by 5
  4. 2023Modelling low-resource accents without accent-specific TTS frontendcited by 3
  5. 2023Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speechcited by 0