mlsanityData to 14 Oct 2026

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

arXiv 2012.15613 ↗First posted 31 Dec 2020cs.CLcited by 318

Not on the map. Why?

Abstract (arXiv)

In this work, we provide a systematic and comprehensive empirical comparison of pretrained multilingual language models versus their monolingual counterparts with regard to their monolingual task performance. We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We first aim to establish, via fair and controlled comparisons, if a gap between the multilingual and the corresponding monolingual representation of that language exists, and subsequently investigate the reason for any performance difference. To disentangle conflating factors, we train new monolingual models on the same data, with monolingually and multilingually trained tokenizers. We find that while the pretraining data size is an important factor, a designated monolingual tokenizer plays an equally important role in the downstream performance. Our results show that languages that are adequately represented in the multilingual model's vocabulary exhibit negligible performance decreases over their monolingual counterparts. We further find that replacing the original multilingual tokenizer with the specialized monolingual tokenizer improves the downstream performance of the multilingual model for almost every task and language.

Key facts. Cited by 318 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 24; 2022: 36; 2023: 43; 2024: 53; 2025: 66; 2026: 96.

318 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  3. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  4. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  5. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  6. 2023SantaCoder: don't reach for the stars!cited by 170
  7. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  8. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  9. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  10. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  11. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 119
  12. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
38 more
  1. 2021KLUE: Korean Language Understanding Evaluationcited by 114
  2. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  3. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  4. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  5. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  6. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 85
  7. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  8. 2025Do Multilingual LLMs Think In English?cited by 78
  9. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  10. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  11. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  12. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 63
  13. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  14. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  15. 2022Language Modelling with Pixelscited by 55
  16. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 55
  17. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 53
  18. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  19. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  20. 2022TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twittercited by 45
  21. 2023Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generationcited by 42
  22. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 41
  23. 2022What do tokens know about their characters and how do they know it?cited by 36
  24. 2023Comparing Biases and the Impact of Multilingual Training across Multiple Languagescited by 36
  25. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 35
  26. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  27. 2023Training-free Lexical Backdoor Attacks on Language Modelscited by 31
  28. 2024On the Challenges and Opportunities in Generative AIcited by 31
  29. 2024Zero-Shot Tokenizer Transfercited by 31
  30. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  31. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  32. 2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
  33. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  34. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  35. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  36. 2022Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languagescited by 25
  37. 2023Cabrita: closing the gap for foreign languagescited by 25
  38. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 25

50 most cited of 318 All 318 (JSON)