mlsanityData to 7 Oct 2026

A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages

arXiv 2403.00686 ↗First posted 1 Mar 2024cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

How should text dataset sizes be compared across languages? Even for content-matched (parallel) corpora, UTF-8 encoded text can require a dramatically different number of bytes for different languages. In our work, we define the byte premium between two languages as the ratio of bytes used to encode content-matched text in those languages. We compute byte premiums for 1155 languages, and we use linear regressions to estimate byte premiums for other languages. We release a tool to obtain byte premiums for any two languages, enabling comparisons of dataset sizes across languages for more equitable multilingual model development and data practices.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 9 of its 22 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

9 of 22 references.

References · 9

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  6. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  7. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  8. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  9. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 13.

22 papers.

Sort:

  1. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  2. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  3. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  4. 2025Confounding Factors in Relating Model Performance to Morphologycited by 14
  5. 2025BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenizationcited by 11
  6. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 10
  7. 2026BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshopcited by 10
  8. 2025BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Datacited by 9
  9. 2025Explaining and Mitigating Crosslingual Tokenizer Inequitiescited by 7
  10. 2025Overcoming Vocabulary Constraints with Pixel-level Fallbackcited by 6
  11. 2025Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratingscited by 3
  12. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
10 more
  1. 2026Tokenization with Split Treescited by 2
  2. 2026Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabetcited by 2
  3. 2026Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Modelscited by 0
  4. 2026Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Modelscited by 0
  5. 2026What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?cited by 0
  6. 2026The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipelinecited by 0
  7. 2026Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptationcited by 0
  8. 2026MoirfEolas and Cr'iochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphologycited by 0
  9. 2026Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergencecited by 0
  10. 2026Cross-Linguistic Effects in Bilingual Phoneme BabyLMscited by 0