# Lifecycle-Optimal Tokenization: Vocabulary Size as a Deployment-Regime-Dependent Infrastructure Parameter

arXiv: https://arxiv.org/abs/2608.11361 · HTML: https://mlsanity.com/paper/2608.11361/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 12 references that are in the map.
- First posted on arXiv 11 Aug 2026.

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://mlsanity.com/paper/2407.13623/) (2024)
- [Getting the most out of your tokenizer for pre-training and domain adaptation](https://mlsanity.com/paper/2402.01035/) (2024)
- [Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs](https://mlsanity.com/paper/2602.10377/) (2026)
- [Compute Optimal Tokenization](https://mlsanity.com/paper/2605.01188/) (2026)
- [Length-MAX Tokenizer for Language Models](https://mlsanity.com/paper/2511.20849/) (2025)

## Abstract (arXiv)

Tokenizer vocabulary size is a foundational design choice in large language model (LLM) infrastructure, yet it is typically fixed at training time based on convention rather than deployment analysis. We show that the cost-optimal vocabulary is not a constant but a function of the serving regime. We formalize total deployment cost as Clifecycle(V) = Ctrain(V) + λ · Cinfer(V, B), where λ is inference volume and B is the serving batch size. Through controlled experiments on two GPU families spanning the memory-bound to compute-bound regimes (A10G, ridge ≈ 117 FLOP/byte; A100, ridge ≈ 183 FLOP/byte), we demonstrate: (1) the inference-optimal vocabulary shifts 16x with serving batch, from 32k at B=1 to 524k at B=64+, driven by amortization of the V × d unembedding matrix read; (2) at 1.3-2.3B model scale, quality (bits per byte, BPB) is optimized at V=65k, confirming scale-dependent vocabulary preference; (3) the lifecycle-optimal vocabulary diverges from training-optimal by up to 16x for production deployments. Quality is approximately invariant across the optimal range (<2% BPB spread), making vocabulary a pure systems optimization with no quality penalty in the measured range. Our results provide actionable capacity planning guidance: on-device deployments (B=1) should use V ≈ 32k; datacenter serving (B ≥ 64, λ ≥ 10) should use V ≈ 131-262k.