mlsanityData to 7 Oct 2026

Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet

arXiv 2606.20993 ↗First posted 18 Jun 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage. Widely-used subword tokenization approaches favor high-resource languages, and tokenizer-free methods still yield longer sequences for scripts with a higher bytes-per-character ratio. To address these shortcomings, we propose to use the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers. IPA provides a compact symbol inventory, greater cross-lingual character overlap, and a more balanced byte-per-character distribution across languages. We train matched pairs of text vs. IPA subword tokenizers across 24 languages and 14 scripts and demonstrate that IPA tokenizers consistently improve tokenization quality, especially for non-Latin scripts, and generalize more effectively to unseen languages and scripts.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 67 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

45 of 67 references.

References · 45

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  9. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  10. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  11. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  12. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
33 more
  1. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  2. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  3. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  4. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  5. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  6. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  7. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  8. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  9. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  10. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  11. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  12. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  13. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  14. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  15. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  16. 2024Toward a Theory of Tokenization in LLMscited by 28
  17. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 27
  18. 2018How Much Does Tokenization Affect Neural Machine Translation?cited by 26
  19. 2024A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languagescited by 22
  20. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 20
  21. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 18
  22. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 15
  23. 2025The Token Tax: Systematic Bias in Multilingual Tokenizationcited by 13
  24. 2023Effects of sub-word segmentation on performance of transformer language modelscited by 12
  25. 2024From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemescited by 11
  26. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  27. 2025False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Modelscited by 11
  28. 2022What Makes for Good Tokenizers in Vision Transformer?cited by 9
  29. 2024When Every Token Counts: Optimal Segmentation for Low-Resource Language Modelscited by 8
  30. 2025FLEXITOKENS: Flexible Tokenization for Evolving Language Modelscited by 8
  31. 2024Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languagescited by 6
  32. 2023IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretrainingcited by 4
  33. 2025Cross-Lingual IPA Contrastive Learning for Zero-Shot NERcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthographycited by 0
  2. 2026Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretrainingcited by 0