mlsanityData to 7 Oct 2026

Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models

arXiv 2305.13707 ↗First posted 23 May 2023cs.CLcited by 186

Not on the map. Why?

Abstract (arXiv)

Language models have graduated from being research prototypes to commercialized products offered as web APIs, and recent works have highlighted the multilingual capabilities of these products. The API vendors charge their users based on usage, more specifically on the number of ``tokens'' processed or generated by the underlying language models. What constitutes a token, however, is training data and model dependent with a large variance in the number of tokens required to convey the same information in different languages. In this work, we analyze the effect of this non-uniformity on the fairness of an API's pricing policy across languages. We conduct a systematic analysis of the cost and utility of OpenAI's language model API on multilingual benchmarks in 22 typologically diverse languages. We show evidence that speakers of a large number of the supported languages are overcharged while obtaining poorer results. These speakers tend to also come from regions where the APIs are less affordable to begin with. Through these analyses, we aim to increase transparency around language model APIs' pricing policies and encourage the vendors to make them more equitable.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). Builds on 61 of its 81 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

61 of 81 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
38 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  4. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  6. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  7. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  8. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  9. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  10. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  11. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  12. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  13. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  14. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  15. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  16. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  17. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  18. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  19. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  20. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  21. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  22. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  23. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  24. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  25. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  26. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  27. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  28. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  29. 2019Multilingual is not enough: BERT for Finnishcited by 118
  30. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  31. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  32. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  33. 2021X-FACT: A New Benchmark Dataset for Multilingual Fact Checkingcited by 90
  34. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  35. 2020Fast WordPiece Tokenizationcited by 85
  36. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
  37. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  38. 2021On Measures of Biases and Harms in NLPcited by 81

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 50; 2025: 53; 2026: 73.

186 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  3. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  4. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  5. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  6. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  7. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  8. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  9. 2025SuperBPE: Space Travel for Language Modelscited by 58
  10. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 55
  11. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  12. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
38 more
  1. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  2. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  3. 2024Natural Language Processing for Dialects of a Language: A Surveycited by 40
  4. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  5. 2024Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personascited by 36
  6. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  7. 2024Stereotype or Personalization? User Identity Biases Chatbot Recommendationscited by 33
  8. 2024On the Challenges and Opportunities in Generative AIcited by 31
  9. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  10. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  11. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  12. 2024Zero-Shot Tokenizer Transfercited by 30
  13. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 27
  14. 2023ChatGPT MT: Competitive for High- (but not Low-) Resource Languagescited by 23
  15. 2023Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyondcited by 22
  16. 2024A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languagescited by 22
  17. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  18. 2024Exploring Design Choices for Building Language-Specific LLMscited by 22
  19. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  20. 2023A Material Lens on Coloniality in NLPcited by 19
  21. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
  22. 2024HyperCLOVA X Technical Reportcited by 18
  23. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  24. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  25. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 18
  26. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledgecited by 16
  27. 2023GlotScript: A Resource and Tool for Low Resource Writing System Identificationcited by 15
  28. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 15
  29. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  30. 2024Near to Mid-term Risks and Opportunities of Open-Source Generative AIcited by 13
  31. 2024Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?cited by 13
  32. 2025Thinking beyond the anthropomorphic paradigm benefits LLM researchcited by 13
  33. 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
  34. 2025The Token Tax: Systematic Bias in Multilingual Tokenizationcited by 13
  35. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafterscited by 12
  36. 2024A Principled Framework for Evaluating on Typologically Diverse Languagescited by 12
  37. 2024Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluationcited by 12
  38. 2025Tokenization is Sensitive to Language Variationcited by 12

50 most cited of 186 All 186 (JSON)