mlsanityData to 7 Oct 2026

Efficient softmax approximation for GPUs

arXiv 1609.04309 ↗First posted 14 Sep 2016cs.CL cs.LGcited by 67

Not on the map. Why?

Abstract (arXiv)

We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computation time. Our approach further reduces the computational time by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax. The code of our method is available at https://github.com/facebookresearch/adaptive-softmax.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 11; 2023: 8; 2024: 11; 2025: 12; 2026: 11.

67 papers.

Sort:

By area:

  1. 2021Hash Layers For Large Sparse Modelscited by 236
  2. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  3. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  4. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  5. 2021Adaptive Semiparametric Language Modelscited by 83
  6. 2021Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classificationcited by 66
  7. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  8. 2022Nonparametric Masked Language Modelingcited by 39
  9. 2024Cut Your Losses in Large-Vocabulary Language Modelscited by 35
  10. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  11. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  12. 2023Why do Nearest Neighbor Language Models Work?cited by 29
38 more
  1. 2021Optimization with Constraint Learning: A Framework and Surveycited by 28
  2. 2022Relational Memory Augmented Language Modelscited by 24
  3. 2022Meta-Learning Fast Weight Language Modelscited by 20
  4. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  5. 2025Scaling Embedding Layers in Language Modelscited by 19
  6. 2023Massively Scaling Heteroscedastic Classifierscited by 11
  7. 2024Large Vocabulary Size Improves Large Language Modelscited by 11
  8. 2021Revisiting Simple Neural Probabilistic Language Modelscited by 10
  9. 2021Distantly Supervised Relation Extraction with Sentence Reconstruction and Knowledge Base Priorscited by 8
  10. 2024An Analysis of BPE Vocabulary Trimming in Neural Machine Translationcited by 7
  11. 2025CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inferencecited by 7
  12. 2022Better Language Model with Hypernym Class Predictioncited by 5
  13. 2025v-PuNNs: van der Put Neural Networks for Transparent Ultrametric Representation Learningcited by 5
  14. 2021Autoencoders as Tools for Program Synthesiscited by 4
  15. 2022Implicit N-grams Induced by Recurrencecited by 4
  16. 2022Stable Invariant Models via Koopman Spectracited by 4
  17. 2023The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensemblescited by 4
  18. 2024A Walsh Hadamard Derived Linear Vector Symbolic Architecturecited by 4
  19. 2025Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generationcited by 4
  20. 2026Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modelingcited by 4
  21. 2021Prototype Memory for Large-scale Face Representation Learningcited by 3
  22. 2021Bag of Tricks for Optimizing Transformer Efficiencycited by 3
  23. 2022Online Trajectory Prediction for Metropolitan Scale Mobility Digital Twincited by 3
  24. 2023ViR: Towards Efficient Vision Retention Backbonescited by 2
  25. 2024Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Modelscited by 2
  26. 2024NLP-enabled Trajectory Map-matching in Urban Road Networks using a Transformer-based Encoder-decodercited by 2
  27. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
  28. 2024Scaling the Vocabulary of Non-autoregressive Models for Efficient Generative Retrievalcited by 2
  29. 2024Optimized Speculative Sampling for GPU Hardware Acceleratorscited by 2
  30. 2025Scalable Trajectory-User Linking with Dual-Stream Representation Networkscited by 2
  31. 2025VQ-Logits: Compressing the Output Bottleneck of Large Language Models via Vector Quantized Logitscited by 2
  32. 2025Multimodal Regression for Enzyme Turnover Rates Predictioncited by 2
  33. 2026BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_basecited by 2
  34. 2021Regularized Training of Nearest Neighbor Language Modelscited by 1
  35. 2022Fast Vocabulary Projection Method via Clustering for Multilingual Machine Translation on GPUcited by 1
  36. 2022SPOT: Knowledge-Enhanced Language Representations for Information Extractioncited by 1
  37. 2022Trajectory-User Linking Is Easier Than You Thinkcited by 1
  38. 2023MobileNMT: Enabling Translation in 15MB and 30mscited by 1

50 most cited of 67 All 67 (JSON)