mlsanityData to 7 Oct 2026

LLM Vocabulary Compression for Low-Compute Environments

arXiv 2411.06371 ↗First posted 10 Nov 2024cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We present a method to compress the final linear layer of language models, reducing memory usage by up to 3.4x without significant performance loss. By grouping tokens based on Byte Pair Encoding (BPE) merges, we prevent materialization of the memory-intensive logits tensor. Evaluations on the TinyStories dataset show that our method performs on par with GPT-Neo and GPT2 while significantly improving throughput by up to 3x, making it suitable for low-compute environments.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 6 of its 9 references that are in the map. First posted on arXiv 10 Nov 2024.

Builds on

6 of 9 references.

References · 6

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  3. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  4. 2016Exploring the Limits of Language Modelingcited by 172
  5. 2016Efficient softmax approximation for GPUscited by 67
  6. 2024The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?cited by 20

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025SpeLLM: Character-Level Multi-Head Decodingcited by 1