# Removable and Irreducible: A Token-Cost Ledger for the Multilingual Tokenization Tax

arXiv: https://arxiv.org/abs/2609.00378 · HTML: https://mlsanity.com/paper/2609.00378/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 17 Jul 2026.

## Builds on (most cited first, 13 of 13)

- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [Byte Latent Transformer: Patches Scale Better Than Tokens](https://mlsanity.com/paper/2412.09871/) (2024)
- [Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://mlsanity.com/paper/2407.13623/) (2024)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [Tokenization and the Noiseless Channel](https://mlsanity.com/paper/2306.16842/) (2023)
- [Toward a Theory of Tokenization in LLMs](https://mlsanity.com/paper/2404.08335/) (2024)
- [The Foundations of Tokenization: Statistical and Computational Concerns](https://mlsanity.com/paper/2407.11606/) (2024)
- [The Token Tax: Systematic Bias in Multilingual Tokenization](https://mlsanity.com/paper/2509.05486/) (2025)
- [MUTANT: A Recipe for Multilingual Tokenizer Design](https://mlsanity.com/paper/2511.03237/) (2025)
- [Compute Optimal Tokenization](https://mlsanity.com/paper/2605.01188/) (2026)
- [Clarifying orthography: Orthographic transparency as compressibility](https://mlsanity.com/paper/2505.13657/) (2025)

## Abstract (arXiv)

Large language models pay a well-documented tax on non-English text: the same content costs several times more tokens, and because attention is quadratic in sequence length, far more compute. We ask how much of this tax is removable. Framing the token layer as source coding -- transformer compute is monotone in sequence length, whose per-atom floor is the Shannon rate H/log2 V, an object already applied to tokenizers in prior work -- we assemble a token-cost ledger that splits each language's cost, at fixed parallel content, into a removable coding redundancy, a residual coding slack, an intrinsic-content term, and an orthogonal, irreducible grapheme-to-phoneme term that governs the multimodal rather than the text cost. On FLORES-200 across eight languages, a production tokenizer costs up to 8.9× more tokens for Indic scripts than for English; a script-matched code trained on 1,012 sentences removes a median 64% of that excess (bootstrap 95% CI [0.638, 0.647]), and a script-fair information floor shows the intrinsic content differs by under 6% -- the tax is representational, not informational. A constructed code removes 98% of a controlled source's redundancy, and the token tax implies up to 79× attention cost. We are explicit about scope and failure: this is compute-and-memory accounting, not a model-quality claim; we neither measure nor claim the cross-lingual direction of the orthographic term; and our matched code is a conservative small-data demonstration. We contribute the unifying ledger, the removable-versus-intrinsic attribution, and an open one-command harness.