# The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics

arXiv: https://arxiv.org/abs/2311.09709 · HTML: https://mlsanity.com/paper/2311.09709/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca](https://mlsanity.com/paper/2309.08958/) (2023)
- [Load What You Need: Smaller Versions of Multilingual BERT](https://mlsanity.com/paper/2010.05609/) (2020)
- [The Highs and Lows of Simple Lexical Domain Adaptation Approaches for Neural Machine Translation](https://mlsanity.com/paper/2101.00421/) (2021)
- [The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation](https://mlsanity.com/paper/2205.06618/) (2022)

## Built on by (most cited first, 3 of 3)

- [AdaptBPE: From General Purpose to Specialized Tokenizers](https://mlsanity.com/paper/2601.21665/) (2026)
- [Lossless Vocabulary Reduction for Auto-Regressive Language Models](https://mlsanity.com/paper/2510.08102/) (2025)
- [COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens](https://mlsanity.com/paper/2509.06836/) (2025)

## Abstract (arXiv)

Deploying large language models (LLMs) encounters challenges due to intensive computational and memory requirements. Our research examines vocabulary trimming (VT) inspired by restricting embedding entries to the language of interest to bolster time and memory efficiency. While such modifications have been proven effective in tasks like machine translation, tailoring them to LLMs demands specific modifications given the diverse nature of LLM applications. We apply two language heuristics to trim the full vocabulary - Unicode-based script filtering and corpus-based selection - to different LLM families and sizes. The methods are straightforward, interpretable, and easy to implement. It is found that VT reduces the memory usage of small models by nearly 50% and has an upper bound of 25% improvement in generation speed. Yet, we reveal the limitations of these methods in that they do not perform consistently well for each language with diminishing returns in larger models.