# Efficient softmax approximation for GPUs

arXiv: https://arxiv.org/abs/1609.04309 · HTML: https://mlsanity.com/paper/1609.04309/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2016.

## Built on by (most cited first, 30 of 67)

- [Hash Layers For Large Sparse Models](https://mlsanity.com/paper/2106.04426/) (2021)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models](https://mlsanity.com/paper/2301.10472/) (2023)
- [Do Transformer Modifications Transfer Across Implementations and Applications?](https://mlsanity.com/paper/2102.11972/) (2021)
- [Adaptive Semiparametric Language Models](https://mlsanity.com/paper/2102.02557/) (2021)
- [Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification](https://mlsanity.com/paper/2110.00685/) (2021)
- [Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval](https://mlsanity.com/paper/2201.12431/) (2022)
- [Nonparametric Masked Language Modeling](https://mlsanity.com/paper/2212.01349/) (2022)
- [Cut Your Losses in Large-Vocabulary Language Models](https://mlsanity.com/paper/2411.09009/) (2024)
- [When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute](https://mlsanity.com/paper/2102.12459/) (2021)
- [Not All Memories are Created Equal: Learning to Forget by Expiring](https://mlsanity.com/paper/2105.06548/) (2021)
- [Why do Nearest Neighbor Language Models Work?](https://mlsanity.com/paper/2301.02828/) (2023)
- [Optimization with Constraint Learning: A Framework and Survey](https://mlsanity.com/paper/2110.02121/) (2021)
- [Relational Memory Augmented Language Models](https://mlsanity.com/paper/2201.09680/) (2022)
- [Meta-Learning Fast Weight Language Models](https://mlsanity.com/paper/2212.02475/) (2022)
- [Scaling Embedding Layers in Language Models](https://mlsanity.com/paper/2502.01637/) (2025)
- [FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling](https://mlsanity.com/paper/2502.14856/) (2025)
- [Massively Scaling Heteroscedastic Classifiers](https://mlsanity.com/paper/2301.12860/) (2023)
- [Large Vocabulary Size Improves Large Language Models](https://mlsanity.com/paper/2406.16508/) (2024)
- [Revisiting Simple Neural Probabilistic Language Models](https://mlsanity.com/paper/2104.03474/) (2021)
- [Distantly Supervised Relation Extraction with Sentence Reconstruction and Knowledge Base Priors](https://mlsanity.com/paper/2104.08225/) (2021)
- [An Analysis of BPE Vocabulary Trimming in Neural Machine Translation](https://mlsanity.com/paper/2404.00397/) (2024)
- [CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference](https://mlsanity.com/paper/2511.21702/) (2025)
- [Better Language Model with Hypernym Class Prediction](https://mlsanity.com/paper/2203.10692/) (2022)
- [v-PuNNs: van der Put Neural Networks for Transparent Ultrametric Representation Learning](https://mlsanity.com/paper/2508.01010/) (2025)
- [Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation](https://mlsanity.com/paper/2511.19835/) (2025)
- [Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling](https://mlsanity.com/paper/2601.12145/) (2026)
- [Autoencoders as Tools for Program Synthesis](https://mlsanity.com/paper/2108.07129/) (2021)
- [Implicit N-grams Induced by Recurrence](https://mlsanity.com/paper/2205.02724/) (2022)
- [Stable Invariant Models via Koopman Spectra](https://mlsanity.com/paper/2207.07475/) (2022)

## Abstract (arXiv)

We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computation time. Our approach further reduces the computational time by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax. The code of our method is available at https://github.com/facebookresearch/adaptive-softmax.