# Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesis

arXiv: https://arxiv.org/abs/2103.03541 · HTML: https://mlsanity.com/paper/2103.03541/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes](https://mlsanity.com/paper/1811.09021/) (2018)
- [How Language-Neutral is Multilingual BERT?](https://mlsanity.com/paper/1911.03310/) (2019)
- [LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition](https://mlsanity.com/paper/2008.03687/) (2020)
- [MultiSpeech: Multi-Speaker Text to Speech with Transformer](https://mlsanity.com/paper/2006.04664/) (2020)
- [Almost Unsupervised Text to Speech and Automatic Speech Recognition](https://mlsanity.com/paper/1905.06791/) (2019)
- [Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis](https://mlsanity.com/paper/1808.10128/) (2018)
- [End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning](https://mlsanity.com/paper/1904.06508/) (2019)
- [One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech](https://mlsanity.com/paper/2008.00768/) (2020)
- [Probing Multilingual BERT for Genetic and Typological Signals](https://mlsanity.com/paper/2011.02070/) (2020)
- [Phonological Features for 0-shot Multilingual Speech Synthesis](https://mlsanity.com/paper/2008.04107/) (2020)
- [Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning](https://mlsanity.com/paper/1910.12729/) (2019)
- [Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languages](https://mlsanity.com/paper/2008.04549/) (2020)
- [Unsupervised Polyglot Text To Speech](https://mlsanity.com/paper/1902.02263/) (2019)
- [Efficient neural speech synthesis for low-resource languages through multilingual modeling](https://mlsanity.com/paper/2008.09659/) (2020)
- [Cross-lingual Multispeaker Text-to-Speech under Limited-Data Scenario](https://mlsanity.com/paper/2005.10441/) (2020)
- [Investigation of learning abilities on linguistic features in sequence-to-sequence text-to-speech synthesis](https://mlsanity.com/paper/2005.10390/) (2020)

## Built on by (most cited first, 13 of 13)

- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech](https://mlsanity.com/paper/2210.15447/) (2022)
- [Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2308.01831/) (2023)
- [SQuId: Measuring Speech Naturalness in Many Languages](https://mlsanity.com/paper/2210.06324/) (2022)
- [Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining](https://mlsanity.com/paper/2301.12596/) (2023)
- [Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings](https://mlsanity.com/paper/2407.14056/) (2024)
- [X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning](https://mlsanity.com/paper/2605.05611/) (2026)
- [Neural Lexicon Reader: Reduce Pronunciation Errors in End-to-end TTS by Leveraging External Textual Knowledge](https://mlsanity.com/paper/2110.09698/) (2021)
- [A multilingual training strategy for low resource Text to Speech](https://mlsanity.com/paper/2409.01217/) (2024)
- [Towards Lifelong Learning of Multilingual Text-To-Speech Synthesis](https://mlsanity.com/paper/2110.04482/) (2021)
- [Few-Shot Cross-Lingual TTS Using Transferable Phoneme Embedding](https://mlsanity.com/paper/2206.15427/) (2022)
- [Towards High-Quality Neural TTS for Low-Resource Languages by Learning Compact Speech Representations](https://mlsanity.com/paper/2210.15131/) (2022)
- [Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initialization](https://mlsanity.com/paper/2402.01692/) (2024)

## Abstract (arXiv)

To scale neural speech synthesis to various real-world languages, we present a multilingual end-to-end framework that maps byte inputs to spectrograms, thus allowing arbitrary input scripts. Besides strong results on 40+ languages, the framework demonstrates capabilities to adapt to new languages under extreme low-resource and even few-shot scenarios of merely 40s transcribed recording, without the need of per-language resources like lexicon, extra corpus, auxiliary models, or linguistic expertise, thus ensuring scalability. While it retains satisfactory intelligibility and naturalness matching rich-resource models. Exhaustive comparative and ablation studies are performed to reveal the potential of the framework for low-resource languages. Furthermore, we propose a novel method to extract language-specific sub-networks in a multilingual model for a better understanding of its mechanism.