# LLM Vocabulary Compression for Low-Compute Environments

arXiv: https://arxiv.org/abs/2411.06371 · HTML: https://mlsanity.com/paper/2411.06371/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 9 references that are in the map.
- First posted on arXiv 10 Nov 2024.

## Builds on (most cited first, 6 of 6)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [Exploring the Limits of Language Modeling](https://mlsanity.com/paper/1602.02410/) (2016)
- [Efficient softmax approximation for GPUs](https://mlsanity.com/paper/1609.04309/) (2016)
- [The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?](https://mlsanity.com/paper/2401.02452/) (2024)

## Built on by (most cited first, 1 of 1)

- [SpeLLM: Character-Level Multi-Head Decoding](https://mlsanity.com/paper/2507.16323/) (2025)

## Abstract (arXiv)

We present a method to compress the final linear layer of language models, reducing memory usage by up to 3.4x without significant performance loss. By grouping tokens based on Byte Pair Encoding (BPE) merges, we prevent materialization of the memory-intensive logits tensor. Evaluations on the TinyStories dataset show that our method performs on par with GPT-Neo and GPT2 while significantly improving throughput by up to 3x, making it suitable for low-compute environments.