Lossless Vocabulary Reduction for Auto-Regressive Language Models
Not on the map. Why?
Abstract (arXiv)
Tokenization -- the process of decomposing a given text into a sequence of subwords called tokens -- is one of the key components in the development of language models. Particularly, auto-regressive language models generate texts token by token, i.e., by predicting the next-token distribution given the previous ones, and thus tokenization directly affects their efficiency in text generation. Since each language model has their own vocabulary as a set of possible tokens, they struggle to cooperate with each other at the level of next-token distributions such as model ensemble. In this paper, we establish a theoretical framework of lossless vocabulary reduction, which efficiently converts a given auto-regressive language model into the one with an arbitrarily small vocabulary without any loss in accuracy. This framework allows language models with different tokenization to cooperate with each other efficiently by reduction to their maximal common vocabulary. Specifically, we empirically demonstrate its applicability to model ensemble with different tokenization.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 45 references that are in the map. First posted on arXiv 9 Oct 2025.
Builds on
30 of 45 references.
References · 30
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
- 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
- 2024Yi: Open Foundation Models by 01.AIcited by 706
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
18 more
- 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
- 2024Knowledge Fusion of Large Language Modelscited by 110
- 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66
- 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
- 2024Bridging the Gap between Different Vocabularies for LLM Ensemblecited by 44
- 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
- 2024ACPBench: Reasoning about Action, Change, and Planningcited by 37
- 2024Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensemblingcited by 33
- 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 30
- 2024From Language Models over Tokens to Language Models over Characterscited by 28
- 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 27
- 2017Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to Englishcited by 23
- 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 19
- 2024Understanding and Mitigating Tokenization Bias in Language Modelscited by 18
- 2024Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensemblescited by 15
- 2024Cool-Fusion: Fuse Large Language Models without Trainingcited by 12
- 2025Sampling from Your Language Model One Byte at a Timecited by 7
- 2023The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristicscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.