mlsanityData to 14 Oct 2026

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization

arXiv 2508.04796 ↗First posted 6 Aug 2025cs.CL cs.AI cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines. Standard algorithms for learning tokenizers rely on frequency-based objectives, which favor languages dominant in the training data and consequently leave lower-resource languages with tokenizations that are disproportionately longer, morphologically implausible, or even riddled with <UNK> placeholders. This phenomenon ultimately amplifies computational and financial inequalities between users from different language backgrounds. To remedy this, we introduce Parity-aware Byte Pair Encoding (BPE), a variant of the widely-used BPE algorithm. At every merge step, Parity-aware BPE applies a fair-max rule that maximizes the compression gain of the currently worst-compressed language, trading a small amount of global compression for cross-lingual parity. We find empirically that Parity-aware BPE reduces tokenization inequality -- operationalized by the Gini coefficient of per-language token costs -- by up to 89% relative to Classical BPE. This comes with negligible impact on global compression rate and no evidence of systematic degradation in downstream LM performance.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 39 of its 64 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

39 of 64 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  11. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  12. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
27 more
  1. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  2. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  3. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  4. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  5. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  6. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  7. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  8. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  9. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  10. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  11. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  12. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  13. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  14. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  15. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  16. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  17. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 65
  18. 2025SuperBPE: Space Travel for Language Modelscited by 63
  19. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 62
  20. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  21. 2020EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answeringcited by 60
  22. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  23. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  24. 2023A Formal Perspective on Byte-Pair Encodingcited by 36
  25. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  26. 2022How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?cited by 20
  27. 2025EuroLLM-9B: Technical Reportcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 16.

20 papers.

Sort:

By area:

  1. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  2. 2026MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignmentcited by 5
  3. 2026GPUTOK: GPU Accelerated Byte Level BPE Tokenizationcited by 3
  4. 2026How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipelinecited by 3
  5. 2025Camellia: Benchmarking Cultural Biases in LLMs for Asian Languagescited by 2
  6. 2026Tokenization with Split Treescited by 2
  7. 2026Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabetcited by 2
  8. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  9. 2025Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmeticcited by 1
  10. 2025Back to Bytes: Revisiting Tokenization Through UTF-8cited by 0
  11. 2025Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performancecited by 0
  12. 2026Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluationcited by 0
8 more
  1. 2026Does Visual Rendering Bypass Tokenization? Investigating Script-Tokenizer Misalignment in Pixel-Based Language Modelscited by 0
  2. 2026From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attributioncited by 0
  3. 2026Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Modelscited by 0
  4. 2026LangMAP: A Language-Adaptive Approach to Tokenizationcited by 0
  5. 2026When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languagescited by 0
  6. 2026To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMscited by 0
  7. 2026Language-Conditioned Token and Reasoning Efficiency in Large Language Models: A Paired Cross-Lingual Study Protocolcited by 0
  8. 2026zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Losscited by 0