# XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech

arXiv: https://arxiv.org/abs/2305.19709 · HTML: https://mlsanity.com/paper/2305.19709/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020)
- [PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS](https://mlsanity.com/paper/2103.15060/) (2021)
- [VnCoreNLP: A Vietnamese Natural Language Processing Toolkit](https://mlsanity.com/paper/1801.01331/) (2018)
- [ByT5 model for massively multilingual grapheme-to-phoneme conversion](https://mlsanity.com/paper/2204.03067/) (2022)
- [Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesis](https://mlsanity.com/paper/2011.05161/) (2020)
- [Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions](https://mlsanity.com/paper/2301.08810/) (2023)
- [A Fast and Accurate Vietnamese Word Segmenter](https://mlsanity.com/paper/1709.06307/) (2017)
- [Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech](https://mlsanity.com/paper/2203.17190/) (2022)
- [DRSpeech: Degradation-Robust Text-to-Speech Synthesis with Frame-Level and Utterance-Level Acoustic Representation Learning](https://mlsanity.com/paper/2203.15683/) (2022)

## Built on by (most cited first, 16 of 16)

- [LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM](https://mlsanity.com/paper/2503.04724/) (2025)
- [Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer](https://mlsanity.com/paper/2402.14279/) (2024)
- [Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamas](https://mlsanity.com/paper/2410.01487/) (2024)
- [An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios](https://mlsanity.com/paper/2406.08911/) (2024)
- [Cross-Lingual IPA Contrastive Learning for Zero-Shot NER](https://mlsanity.com/paper/2503.07214/) (2025)
- [PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding](https://mlsanity.com/paper/2506.11064/) (2025)
- [Whisper based Cross-Lingual Phoneme Recognition between Vietnamese and English](https://mlsanity.com/paper/2508.19270/) (2025)
- [3DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control](https://mlsanity.com/paper/2601.18451/) (2026)
- [Zero-Shot Cross-Lingual NER Using Phonemic Representations for Low-Resource Languages](https://mlsanity.com/paper/2406.16030/) (2024)
- [LAPS-Diff: A Diffusion-Based Framework for Singing Voice Synthesis With Language Aware Prosody-Style Guided Learning](https://mlsanity.com/paper/2507.04966/) (2025)
- [Multi-task Pretraining for Enhancing Interpretable L2 Pronunciation Assessment](https://mlsanity.com/paper/2509.16876/) (2025)
- [UR-BERT: Scaling Text Encoders for Massively Multilingual TTS Through Universal Romanization and Speech Token Prediction](https://mlsanity.com/paper/2606.11681/) (2026)
- [PiDA: Phonetically-Informed Data Augmentation for Robust Vietnamese Speech Translation](https://mlsanity.com/paper/2606.12911/) (2026)
- [Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing](https://mlsanity.com/paper/2608.06424/) (2026)
- [One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography](https://mlsanity.com/paper/2608.25904/) (2026)
- [Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing](https://mlsanity.com/paper/2610.00825/) (2026)

## Abstract (arXiv)

We present XPhoneBERT, the first multilingual model pre-trained to learn phoneme representations for the downstream text-to-speech (TTS) task. Our XPhoneBERT has the same model architecture as BERT-base, trained using the RoBERTa pre-training approach on 330M phoneme-level sentences from nearly 100 languages and locales. Experimental results show that employing XPhoneBERT as an input phoneme encoder significantly boosts the performance of a strong neural TTS model in terms of naturalness and prosody and also helps produce fairly high-quality speech with limited training data. We publicly release our pre-trained XPhoneBERT with the hope that it would facilitate future research and downstream TTS applications for multiple languages. Our XPhoneBERT model is available at https://github.com/VinAIResearch/XPhoneBERT