mlsanityData to 14 Oct 2026

Why Gradients Rapidly Increase Near the End of Training

arXiv 2506.02285 ↗First posted 2 Jun 2025cs.LG cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

During long-duration Large Language Model (LLM) training runs the gradient norm increases rapidly near the end of training. In this short note, we show that this increase is due to an unintended interaction between weight decay, normalization layers, and the learning rate schedule. We propose a simple correction that fixes this behavior while also resulting in lower loss values throughout training.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 16 of its 20 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

16 of 20 references.

References · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Layer Normalizationcited by 5,669
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  7. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
  8. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  9. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  10. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  11. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  12. 2020On the distance between two neural networks and the stability of learningcited by 46
4 more
  1. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  2. 2020On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspectivecited by 25
  3. 2020The Power of Factorial Powers: New Parameter settings for (Stochastic) Optimizationcited by 8
  4. 2019Beyond Folklore: A Scaling Calculus for the Design and Initialization of ReLU Networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 15.

22 papers.

Sort:

By area:

  1. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 16
  2. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 15
  3. 2025Cautious Weight Decaycited by 13
  4. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
  5. 2026Rethinking Language Model Scaling under Transferable Hypersphere Optimizationcited by 12
  6. 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 9
  7. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  8. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 7
  9. 2025Relative Scaling Laws for LLMscited by 6
  10. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 5
  11. 2026Full-bandwidth transformercited by 5
  12. 2025K2-V2: A 360-Open, Reasoning-Enhanced LLMcited by 3
10 more
  1. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  2. 2025Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradientscited by 2
  3. 2026Scale Weight Decay and Train Bettercited by 2
  4. 2025Correction of Decoupled Weight Decaycited by 1
  5. 2026Demystifying Manifold Constraints in LLM Pre-trainingcited by 1
  6. 2026Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memorycited by 1
  7. 2026OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizerscited by 0
  8. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0
  9. 2026When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decaycited by 0
  10. 2026Second-Moment Stochastic Approximation Methodscited by 0