mlsanityData to 14 Oct 2026

Spike No More: Stabilizing the Pre-training of Large Language Models

arXiv 2312.16903 ↗First posted 28 Dec 2023cs.CL cs.AIcited by 52

Not on the map. Why?

Abstract (arXiv)

Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre-training needs a vast computational budget, we should avoid such spikes. Based on the assumption that the loss spike is caused by the sudden growth of the gradient norm, we explore factors to keep the gradient norm small through an analysis of the spectral norms of the Jacobian matrices for the sub-layers. Our findings suggest that stabilizing the pre-training process requires two conditions: small sub-layers and large shortcut. We conduct various experiments to empirically verify our theoretical analyses. Experimental results demonstrate that methods satisfying the conditions effectively prevent loss spikes during pre-training.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

45 of 58 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
33 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2019Root Mean Square Layer Normalizationcited by 1,480
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  8. 2020On Layer Normalization in the Transformer Architecturecited by 922
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  13. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  14. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  15. 2016Using the Output Embedding to Improve Language Modelscited by 272
  16. 2020Understanding the Difficulty of Training Transformerscited by 210
  17. 2018Scaling Neural Machine Translationcited by 204
  18. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  19. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  20. 2015Training Very Deep Networkscited by 173
  21. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  22. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  23. 2023A Spectral Condition for Feature Learningcited by 109
  24. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  25. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 92
  26. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  27. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  28. 2021How to Train BERT with an Academic Budgetcited by 80
  29. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 71
  30. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  31. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 36
  32. 2021Rethinking Perturbations in Encoder-Decoders for Fast Trainingcited by 17
  33. 2022B2T Connection: Serving Stability and Performance in Deep Transformerscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 22; 2026: 22.

52 papers.

Sort:

By area:

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  2. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  3. 2025The Curse of Depth in Large Language Modelscited by 62
  4. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 55
  5. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  6. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  7. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 34
  8. 2026Controlled LLM Training on Spectral Spherecited by 31
  9. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
  10. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 22
  11. 2024Self-attention Networks Localize When QK-eigenspectrum Concentratescited by 19
  12. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoningcited by 15
38 more
  1. 2026Scaling Embeddings Outperforms Scaling Experts in Language Modelscited by 15
  2. 2026Arcee Trinity Large Technical Reportcited by 11
  3. 2025ZClip: Adaptive Spike Mitigation for LLM Pre-Trainingcited by 10
  4. 2024HLAT: High-quality Large Language Model Pre-trained on AWS Trainiumcited by 8
  5. 2025Efficient Construction of Model Family through Progressive Training Using Model Expansioncited by 8
  6. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  7. 2026Loop the Loopies!cited by 8
  8. 2024YuLan-Mini: An Open Data-efficient Language Modelcited by 7
  9. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7
  10. 2024Stable Language Model Pre-training by Reducing Embedding Variabilitycited by 6
  11. 2025AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clippingcited by 6
  12. 2025GradientStabilizer:Fix the Norm, Not the Gradientcited by 6
  13. 2025Pre-Training LLMs on a budget: A comparison of three optimizerscited by 5
  14. 2025When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMscited by 5
  15. 2024Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikescited by 4
  16. 2025Instability in Downstream Task Performance During LLM Pretrainingcited by 4
  17. 2026Output Embedding Centering for Stable LLM Pretrainingcited by 4
  18. 2026When Does Sparsity Mitigate the Curse of Depth in LLMscited by 4
  19. 2026PowLU: An Activation Function for Stable Pre-Training of LLMscited by 4
  20. 2025Controlling changes to attention logitscited by 3
  21. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 3
  22. 2025GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scalingcited by 2
  23. 2025Spectral Alignment as Predictor of Loss Explosion in Neural Network Trainingcited by 2
  24. 2026SpanNorm: Reconciling Training Stability and Performance in Deep Transformerscited by 2
  25. 2026Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Ratescited by 2
  26. 2026Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instabilitycited by 2
  27. 2024A Mechanistic Study of Transformers Training Dynamicscited by 1
  28. 2025Interactive Training: Feedback-Driven Neural Network Optimizationcited by 1
  29. 2025Knocking-Heads Attentioncited by 1
  30. 2026DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providerscited by 1
  31. 2026GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimizationcited by 1
  32. 2024Technical Report: Small Language Model for Japanese Clinical and Medicinecited by 0
  33. 2025Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Trickscited by 0
  34. 2026AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Trainingcited by 0
  35. 2026Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformerscited by 0
  36. 2026Learn-by-Wire Training Control Governance: Bounded Autonomous Training Under Stress for Stability and Efficiencycited by 0
  37. 2026GNMR: Runtime Stability Control for Low-Precision Large Language Model Trainingcited by 0
  38. 2026Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutionscited by 0

50 most cited of 52 All 52 (JSON)