# ByT5 model for massively multilingual grapheme-to-phoneme conversion

arXiv: https://arxiv.org/abs/2204.03067 · HTML: https://mlsanity.com/paper/2204.03067/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 13 of 13)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Transformer based Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/2004.06338/) (2020)
- [Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/1904.03446/) (2019)
- [Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/1506.00196/) (2015)
- [Massively Multilingual Neural Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/1708.01464/) (2017)
- [g2pM: A Neural Grapheme-to-Phoneme Conversion Package for Mandarin Chinese Based on a New Open Benchmark Dataset](https://mlsanity.com/paper/2004.03136/) (2020)
- [Jointly Learning to Align and Convert Graphemes to Phonemes with Neural Attention Models](https://mlsanity.com/paper/1610.06540/) (2016)
- [One Model to Pronounce Them All: Multilingual Grapheme-to-Phoneme Conversion With a Transformer Ensemble](https://mlsanity.com/paper/2006.13343/) (2020)

## Built on by (most cited first, 30 of 49)

- [ACE-Step: A Step Towards Music Generation Foundation Model](https://mlsanity.com/paper/2506.00045/) (2025)
- [ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models](https://mlsanity.com/paper/2212.10474/) (2022)
- [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025)
- [XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech](https://mlsanity.com/paper/2305.19709/) (2023)
- [The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language](https://mlsanity.com/paper/2311.08323/) (2023)
- [POWSM: A Phonetic Open Whisper-Style Speech Foundation Model](https://mlsanity.com/paper/2510.24992/) (2025)
- [LLM-Powered Grapheme-to-Phoneme Conversion: Benchmark and Case Study](https://mlsanity.com/paper/2409.08554/) (2024)
- [Speak While You Think: Streaming Speech Synthesis During Text Generation](https://mlsanity.com/paper/2309.11210/) (2023)
- [SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation](https://mlsanity.com/paper/2310.00796/) (2023)
- [LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM](https://mlsanity.com/paper/2503.04724/) (2025)
- [IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling](https://mlsanity.com/paper/2504.03036/) (2025)
- [LingGym: How Far Are LLMs from Thinking Like Field Linguists?](https://mlsanity.com/paper/2511.00343/) (2025)
- [Exploring the Benefits of Tokenization of Discrete Acoustic Units](https://mlsanity.com/paper/2406.05547/) (2024)
- [PolyIPA -- Multilingual Phoneme-to-Grapheme Conversion Model](https://mlsanity.com/paper/2412.09102/) (2024)
- [PRiSM: Benchmarking Phone Realization in Speech Models](https://mlsanity.com/paper/2601.14046/) (2026)
- [Data-driven grapheme-to-phoneme representations for a lexicon-free text-to-speech](https://mlsanity.com/paper/2401.10465/) (2024)
- [Pronunciation-Lexicon Free Training for Phoneme-based Crosslingual ASR via Joint Stochastic Approximation](https://mlsanity.com/paper/2507.06249/) (2025)
- [CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonese](https://mlsanity.com/paper/2511.04139/) (2025)
- [HuPER: A Human-Inspired Framework for Phonetic Perception](https://mlsanity.com/paper/2602.01634/) (2026)
- [Improving grapheme-to-phoneme conversion by learning pronunciations from speech recordings](https://mlsanity.com/paper/2307.16643/) (2023)
- [Multilingual context-based pronunciation learning for Text-to-Speech](https://mlsanity.com/paper/2307.16709/) (2023)
- [Improving Grapheme-to-Phoneme Conversion through In-Context Knowledge Retrieval with Large Language Models](https://mlsanity.com/paper/2411.07563/) (2024)
- [Mitigating the Exposure Bias in Sentence-Level Grapheme-to-Phoneme (G2P) Transduction](https://mlsanity.com/paper/2308.08442/) (2023)
- [Acquiring Pronunciation Knowledge from Transcribed Speech Audio via Multi-task Learning](https://mlsanity.com/paper/2409.09891/) (2024)
- [Efficiently Identifying Low-Quality Language Subsets in Multilingual Datasets: A Case Study on a Large-Scale Multilingual Audio Dataset](https://mlsanity.com/paper/2410.04292/) (2024)
- [AyutthayaAlpha: A Thai-Latin Script Transliteration Transformer](https://mlsanity.com/paper/2412.03877/) (2024)
- [Cross-Lingual IPA Contrastive Learning for Zero-Shot NER](https://mlsanity.com/paper/2503.07214/) (2025)
- [What You Read Isn't What You Hear: Linguistic Sensitivity in Deepfake Speech Detection](https://mlsanity.com/paper/2505.17513/) (2025)
- [LatPhon: Lightweight Multilingual G2P for Romance Languages and English](https://mlsanity.com/paper/2509.03300/) (2025)
- [ReNikud: Audio-Supervised Hebrew Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/2606.20179/) (2026)

## Abstract (arXiv)

In this study, we tackle massively multilingual grapheme-to-phoneme conversion through implementing G2P models based on ByT5. We have curated a G2P dataset from various sources that covers around 100 languages and trained large-scale multilingual G2P models based on ByT5. We found that ByT5 operating on byte-level inputs significantly outperformed the token-based mT5 model in terms of multilingual G2P. Pairwise comparison with monolingual models in these languages suggests that multilingual ByT5 models generally lower the phone error rate by jointly learning from a variety of languages. The pretrained model can further benefit low resource G2P through zero-shot prediction on unseen languages or provides pretrained weights for finetuning, which helps the model converge to a lower phone error rate than randomly initialized weights. To facilitate future research on multilingual G2P, we make available our code and pretrained multilingual G2P models at: https://github.com/lingjzhu/CharsiuG2P.