mlsanityData to 14 Oct 2026

Language Model Tokenizers Introduce Unfairness Between Languages

arXiv 2305.15425 ↗First posted 17 May 2023cs.CL cs.LGcited by 214

Not on the map. Why?

Abstract (arXiv)

Recent language models have shown impressive multilingual performance, even when not explicitly trained for it. Despite this, there are concerns about the quality of their outputs across different languages. In this paper, we show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked. The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases. These disparities persist even for tokenizers that are intentionally trained for multilingual support. Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs. This induces unfair treatment for some language communities in regard to the cost of accessing commercial language services, the processing time and latency, as well as the amount of content that can be provided as context to the models. Therefore, we make the case that we should train future language models using multilingually fair subword tokenizers.

Key facts. Cited by 214 papers in the map (data through 14 Oct 2026). Builds on 62 of its 117 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

62 of 117 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
38 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  9. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  10. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  11. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  12. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  13. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  14. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  15. 2019CamemBERT: a Tasty French Language Modelcited by 368
  16. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  17. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  18. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  19. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  20. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  21. 2021MuRIL: Multilingual Representations for Indian Languagescited by 179
  22. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  23. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 176
  24. 2020PhoBERT: Pre-trained language models for Vietnamesecited by 149
  25. 2022On The Computational Complexity of Self-Attentioncited by 139
  26. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  27. 2019Multilingual is not enough: BERT for Finnishcited by 119
  28. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  29. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 109
  30. 2019Discourse-Aware Neural Extractive Text Summarizationcited by 105
  31. 2019Neural Machine Translation: A Review and Surveycited by 100
  32. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  33. 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69
  34. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  35. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 62
  36. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  37. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  38. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 53

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 54; 2025: 63; 2026: 85.

214 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 254
  2. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  3. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 113
  4. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  5. 2025Do Multilingual LLMs Think In English?cited by 78
  6. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 75
  7. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  8. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  9. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 69
  10. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  11. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  12. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 48
38 more
  1. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  2. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45
  3. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 36
  4. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 34
  5. 2023ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Eventscited by 31
  6. 2024On the Challenges and Opportunities in Generative AIcited by 31
  7. 2024Zero-Shot Tokenizer Transfercited by 31
  8. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  9. 2023A Comprehensive Evaluation of Large Language Models on Legal Judgment Predictioncited by 28
  10. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languagescited by 28
  11. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 28
  12. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  13. 2024A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languagescited by 25
  14. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  15. 2024Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMscited by 22
  16. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  17. 2024Where is the signal in tokenization space?cited by 21
  18. 2024SignCLIP: Connecting Text and Sign Language by Contrastive Learningcited by 20
  19. 2024Egalitarian Language Representation in Language Models: It All Begins with Tokenizerscited by 20
  20. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
  21. 2024Poro 34B and the Blessing of Multilingualitycited by 18
  22. 2024HyperCLOVA X Technical Reportcited by 18
  23. 2024Understanding and Mitigating Tokenization Bias in Language Modelscited by 18
  24. 2024Multi-word Tokenization for Sequence Compressioncited by 17
  25. 2025HyperFree: A Channel-adaptive and Tuning-free Foundation Model for Hyperspectral Remote Sensing Imagerycited by 17
  26. 2023GlotScript: A Resource and Tool for Low Resource Writing System Identificationcited by 15
  27. 2024Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Modelscited by 15
  28. 2025The Token Tax: Systematic Bias in Multilingual Tokenizationcited by 14
  29. 2024Near to Mid-term Risks and Opportunities of Open-Source Generative AIcited by 13
  30. 2024Risks and Opportunities of Open-Source Generative AIcited by 13
  31. 2024Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilitiescited by 13
  32. 2025Tokenization is Sensitive to Language Variationcited by 13
  33. 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
  34. 2024Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russiancited by 12
  35. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafterscited by 12
  36. 2024A Principled Framework for Evaluating on Typologically Diverse Languagescited by 12
  37. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Expertscited by 12
  38. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12

50 most cited of 214 All 214 (JSON)