# Lossless Vocabulary Reduction for Auto-Regressive Language Models

arXiv: https://arxiv.org/abs/2510.08102 · HTML: https://mlsanity.com/paper/2510.08102/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 30 of 30)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Knowledge Fusion of Large Language Models](https://mlsanity.com/paper/2401.10491/) (2024)
- [Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration](https://mlsanity.com/paper/2404.12715/) (2024)
- [An Emulator for Fine-Tuning Large Language Models using Small Language Models](https://mlsanity.com/paper/2310.12962/) (2023)
- [Bridging the Gap between Different Vocabularies for LLM Ensemble](https://mlsanity.com/paper/2404.09492/) (2024)
- [Fast Vocabulary Transfer for Language Model Compression](https://mlsanity.com/paper/2402.09977/) (2024)
- [ACPBench: Reasoning about Action, Change, and Planning](https://mlsanity.com/paper/2410.05669/) (2024)
- [Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling](https://mlsanity.com/paper/2406.12585/) (2024)
- [BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training](https://mlsanity.com/paper/2409.04599/) (2024)
- [From Language Models over Tokens to Language Models over Characters](https://mlsanity.com/paper/2412.03719/) (2024)
- [Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling](https://mlsanity.com/paper/2410.03777/) (2024)
- [Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to English](https://mlsanity.com/paper/1707.09879/) (2017)
- [An Efficient Multilingual Language Model Compression through Vocabulary Trimming](https://mlsanity.com/paper/2305.15020/) (2023)
- [Understanding and Mitigating Tokenization Bias in Language Models](https://mlsanity.com/paper/2406.16829/) (2024)
- [Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles](https://mlsanity.com/paper/2410.09303/) (2024)
- [Cool-Fusion: Fuse Large Language Models without Training](https://mlsanity.com/paper/2407.19807/) (2024)
- [Sampling from Your Language Model One Byte at a Time](https://mlsanity.com/paper/2506.14123/) (2025)
- [The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics](https://mlsanity.com/paper/2311.09709/) (2023)

## Built on by (most cited first, 2 of 2)

- [Parallel In-context Learning for Large Vision Language Models](https://mlsanity.com/paper/2603.16092/) (2026)
- [To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs](https://mlsanity.com/paper/2609.15528/) (2026)

## Abstract (arXiv)

Tokenization -- the process of decomposing a given text into a sequence of subwords called tokens -- is one of the key components in the development of language models. Particularly, auto-regressive language models generate texts token by token, i.e., by predicting the next-token distribution given the previous ones, and thus tokenization directly affects their efficiency in text generation. Since each language model has their own vocabulary as a set of possible tokens, they struggle to cooperate with each other at the level of next-token distributions such as model ensemble. In this paper, we establish a theoretical framework of lossless vocabulary reduction, which efficiently converts a given auto-regressive language model into the one with an arbitrarily small vocabulary without any loss in accuracy. This framework allows language models with different tokenization to cooperate with each other efficiently by reduction to their maximal common vocabulary. Specifically, we empirically demonstrate its applicability to model ensemble with different tokenization.