mlsanityData to 14 Oct 2026

Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck

arXiv 2404.07647 ↗First posted 11 Apr 2024cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Recent advances in language modeling consist in pretraining highly parameterized neural networks on extremely large web-mined text corpora. Training and inference with such models can be costly in practice, which incentivizes the use of smaller counterparts. However, it has been observed that smaller models can suffer from saturation, characterized as a drop in performance at some advanced point in training followed by a plateau. In this paper, we find that such saturation can be explained by a mismatch between the hidden dimension of smaller models and the high rank of the target contextual probability distribution. This mismatch affects the performance of the linear prediction head used in such models through the well-known softmax bottleneck phenomenon. We measure the effect of the softmax bottleneck in various settings and find that models based on less than 1000 hidden dimensions tend to adopt degenerate latent representations in late pretraining, which leads to reduced evaluation performance.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

22 of 34 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  7. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  8. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  9. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  10. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  11. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
  12. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 139
10 more
  1. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 63
  2. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  3. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 53
  4. 2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27
  5. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
  6. 2018Sigsoftmax: Reanalysis of the Softmax Bottleneckcited by 20
  7. 2021Frequency-based Distortions in Contextualized Word Embeddingscited by 20
  8. 2022A Natural Bias for Language Generation Modelscited by 6
  9. 2023Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translationcited by 5
  10. 2021Breaking the Softmax Bottleneck for Sequential Recommender Systems with Dropout and Decouplingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 6.

21 papers.

Sort:

By area:

  1. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  2. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 37
  3. 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runscited by 35
  4. 2024Causal Estimation of Memorisation Profilescited by 20
  5. 2025Guiding Reasoning in Small Language Models with LLM Assistancecited by 14
  6. 2025Sequences of Logits Reveal the Low Rank Structure of Language Modelscited by 9
  7. 2026Lost in Backpropagation: The LM Head is a Gradient Bottleneckcited by 9
  8. 2024Tending Towards Stability: Convergence Challenges in Small Language Modelscited by 7
  9. 2026Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamicscited by 6
  10. 2025Pretraining Scaling Laws for Generative Evaluations of Language Modelscited by 5
  11. 2025Learning Dynamics of Meta-Learning in Small Model Pretrainingcited by 3
  12. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 3
9 more
  1. 2024Norm of Mean Contextualized Embeddings Determines their Variancecited by 1
  2. 2024Unsupervised Data Validation Methods for Efficient Model Trainingcited by 1
  3. 2025Leveraging Importance Sampling to Detach Alignment Modules from Large Language Modelscited by 1
  4. 2025Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Modelscited by 1
  5. 2025EasyMath: A 0-shot Math Benchmark for SLMscited by 0
  6. 2025Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Trainingcited by 0
  7. 2026The Geometry of Low-Resource Language Representationscited by 0
  8. 2026The Gradient That Top-k Distillation Never Computescited by 0
  9. 2026Copy-Paste Language Modelscited by 0