mlsanityData to 14 Oct 2026

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

arXiv 1711.03953 ↗First posted 10 Nov 2017cs.CL cs.LGcited by 152

Not on the map. Why?

Abstract (arXiv)

We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embeddings does not have enough capacity to model natural language. We propose a simple and effective method to address this issue, and improve the state-of-the-art perplexities on Penn Treebank and WikiText-2 to 47.69 and 40.68 respectively. The proposed method also excels on the large-scale 1B Word dataset, outperforming the baseline by over 5.6 points in perplexity.

Key facts. Cited by 152 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 20; 2022: 15; 2023: 21; 2024: 31; 2025: 25; 2026: 40.

152 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  3. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  4. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  5. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  6. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  7. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 75
  8. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  9. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54
  10. 2021Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitivescited by 50
  11. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  12. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
38 more
  1. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  2. 2021I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviewscited by 45
  3. 2022Nonparametric Masked Language Modelingcited by 40
  4. 2024Logits of API-Protected LLMs Leak Proprietary Informationcited by 40
  5. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  6. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 37
  7. 2025Language Models are Injective and Hence Invertiblecited by 35
  8. 2023Closing the Curious Case of Neural Text Degenerationcited by 32
  9. 2024Transformers Can Represent n-gram Language Modelscited by 32
  10. 2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimizationcited by 31
  11. 2023Why do Nearest Neighbor Language Models Work?cited by 29
  12. 2025Pre-training under infinite computecited by 29
  13. 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
  14. 2021Selfish Sparse RNN Trainingcited by 26
  15. 2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
  16. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  17. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  18. 2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 21
  19. 2024What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languagescited by 20
  20. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 19
  21. 2022Improving Semantic Matching through Dependency-Enhanced Pre-trained Model with Adaptive Fusioncited by 18
  22. 2024On the Independence Assumption in Neurosymbolic Learningcited by 18
  23. 2021Rethinking Perturbations in Encoder-Decoders for Fast Trainingcited by 17
  24. 2024A Survey on LLM Inference-Time Self-Improvementcited by 17
  25. 2021Which transformer architecture fits my data? A vocabulary bottleneck in self-attentioncited by 16
  26. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 16
  27. 2022Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practicecited by 15
  28. 2022Context Variance Evaluation of Pretrained Language Models for Prompt-based Biomedical Knowledge Probingcited by 15
  29. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  30. 2023Tensor Decomposition for Model Reduction in Neural Networks: A Reviewcited by 14
  31. 2023Revisiting Neural Retrieval on Acceleratorscited by 14
  32. 2026Subliminal Steering: Stronger Encoding of Hidden Signalscited by 14
  33. 2024Banishing LLM Hallucinations Requires Rethinking Generalizationcited by 13
  34. 2023Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classificationcited by 11
  35. 2023Representation Deficiency in Masked Language Modelingcited by 10
  36. 2025Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learningcited by 10
  37. 2025Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanationcited by 9
  38. 2025Better Language Model Inversion by Compactly Representing Next-Token Distributionscited by 9

50 most cited of 152 All 152 (JSON)