mlsanityData to 14 Oct 2026

Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages

arXiv 2305.17179 ↗First posted 26 May 2023cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

Multilingual language models have recently gained attention as a promising solution for representing multiple languages in a single model. In this paper, we propose new criteria to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. Our findings show that the overlap of vocabulary across languages can be actually detrimental to certain downstream tasks (POS, dependency tree labeling). In contrast, NER and sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing vocabulary. We also observe that the coverage of the language-specific tokens in the multilingual vocabulary significantly impacts the word-level tasks. Our study offers a deeper understanding of the role of tokenizers in multilingual language models and guidelines for future model developers to choose the most suitable tokenizer for their specific application before undertaking costly model pre-training

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 8 of its 42 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

8 of 42 references.

References · 8

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  4. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  5. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  6. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  7. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  8. 2022You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 21; 2026: 30.

62 papers.

Sort:

By area:

  1. 2024Tokenization Is More Than Compressioncited by 89
  2. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 69
  3. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  4. 2023Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Promptscited by 45
  5. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  6. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  7. 2023Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologiescited by 16
  8. 2025How Much is Enough? The Diminishing Returns of Tokenization Training Datacited by 15
  9. 2024Lexically Grounded Subword Segmentationcited by 13
  10. 2025Tokenization is Sensitive to Language Variationcited by 13
  11. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  12. 2025False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Modelscited by 11
38 more
  1. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  2. 2025Beyond Literal Token Overlap: Token Alignability for Multilingualitycited by 9
  3. 2025See the Text: From Tokenization to Visual Readingcited by 9
  4. 2025Explaining and Mitigating Crosslingual Tokenizer Inequitiescited by 7
  5. 2024How Multilingual Are Large Language Models Fine-Tuned for Translation?cited by 6
  6. 2024Elucidating the design space of language models for image generationcited by 6
  7. 2025A Partition Cover Approach to Tokenizationcited by 5
  8. 2026The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Modelscited by 4
  9. 2026Compute Optimal Tokenizationcited by 4
  10. 2024Are Data Augmentation Methods in Named Entity Recognition Applicable for Uncertainty Estimation?cited by 3
  11. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
  12. 2024Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in Large Language Modelscited by 2
  13. 2025Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insightscited by 2
  14. 2025Scalable multilingual PII annotation for responsible AI in LLMscited by 2
  15. 2025Happiness is Sharing a Vocabulary: A Study of Transliteration Methodscited by 2
  16. 2025Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiencycited by 2
  17. 2026Form and Meaning in Intrinsic Multilingual Evaluationscited by 2
  18. 2026Language Models are not Equally Robust to Non-Canonical Tokenization across Languagescited by 2
  19. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  20. 2026Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertilitycited by 2
  21. 2025From No to Know: Taxonomy, Challenges, and Opportunities for Negation Understanding in Multimodal Foundation Modelscited by 1
  22. 2025Uncovering inequalities in new knowledge learning by large language models across different languagescited by 1
  23. 2025UniNet: A Unified Multi-granular Traffic Modeling Framework for Network Securitycited by 1
  24. 2025Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmeticcited by 1
  25. 2026Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Featurescited by 1
  26. 2026Semantic Chunking and the Entropy of Natural Languagecited by 1
  27. 2026Pingala: Prosody-Aware Decoding for Sanskrit Poetry Generationcited by 1
  28. 2026Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretrainingcited by 1
  29. 2025Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Datasetcited by 0
  30. 2025Conditional Unigram Tokenization with Parallel Datacited by 0
  31. 2025MoVoC: Morphology-Aware Subword Construction for Geez Script Languagescited by 0
  32. 2025Languages are Modalities: Cross-Lingual Alignment via Encoder Injectioncited by 0
  33. 2026Training Language Models with homotokens Leads to Delayed Overfittingcited by 0
  34. 2026Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluationcited by 0
  35. 2026Does Visual Rendering Bypass Tokenization? Investigating Script-Tokenizer Misalignment in Pixel-Based Language Modelscited by 0
  36. 2026LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptationcited by 0
  37. 2026Toward Culturally Grounded Natural Language Processingcited by 0
  38. 2026An In-Vitro Study on Cross-Lingual Generalization in Language Modelscited by 0

50 most cited of 62 All 62 (JSON)