mlsanityData to 7 Oct 2026

Stable Language Model Pre-training by Reducing Embedding Variability

arXiv 2409.07787 ↗First posted 12 Sep 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Stable pre-training is essential for achieving better-performing language models. However, tracking pre-training stability by calculating gradient variance at every step is impractical due to the significant computational costs. We explore Token Embedding Variability (TEV) as a simple and efficient proxy for assessing pre-training stability in language models with pre-layer normalization, given that shallower layers are more prone to gradient explosion (section 2.2). Moreover, we propose Multi-head Low-Rank Attention (MLRA) as an architecture to alleviate such instability by limiting the exponential growth of output embedding variance, thereby preventing the gradient explosion (section 3.2). Empirical results on GPT-2 with MLRA demonstrate increased stability and lower perplexity, particularly in deeper models.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

39 of 53 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2016Layer Normalizationcited by 5,579
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
27 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  5. 2020On Layer Normalization in the Transformer Architecturecited by 906
  6. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  7. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  8. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  9. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  10. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  11. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  12. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  13. 2016Using the Output Embedding to Improve Language Modelscited by 266
  14. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 251
  15. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  16. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 156
  17. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  18. 2019On Identifiability in Transformerscited by 115
  19. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  20. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  21. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
  22. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  23. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  24. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  25. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
  26. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39
  27. 2020A Study of Gradient Variance in Deep Learningcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runscited by 33
  2. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 32
  3. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  4. 2025Exploiting Vocabulary Frequency Imbalance in Language Model Pre-trainingcited by 7
  5. 2026On the Spectral Flattening of Quantized Embeddingscited by 0
  6. 2026Continuity and Ordinality Matter: Constraining Time Series Tokens for Effective Time Series Analysis with Large Language Modelscited by 0