mlsanityData to 14 Oct 2026

A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs

arXiv 2601.16979 ↗First posted 23 Jan 2026cs.LG cond-mat.dis-nn cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Understanding the curvature evolution of the loss landscape is fundamental to analyzing the training dynamics of neural networks. The most commonly studied measure, Hessian sharpness (λmaxH) -- the largest eigenvalue of the loss Hessian -- determines local training stability and interacts with the learning rate throughout training. Despite its significance in analyzing training dynamics, direct measurement of Hessian sharpness remains prohibitive for Large Language Models (LLMs) due to high computational cost. We analyze textitcritical sharpness (λc), a computationally efficient measure requiring fewer than 10 forward passes given the update direction Delta θ. Critically, this measure captures well-documented Hessian sharpness phenomena, including progressive sharpening and Edge of Stability. Using this measure, we provide the first demonstration of these sharpness phenomena at scale, up to 7B parameters, spanning both pre-training and mid-training of OLMo-2 models. We further introduce textitrelative critical sharpness (λc1→ 2), which quantifies the curvature of one loss landscape while optimizing another, to analyze the transition from pre-training to fine-tuning and guide data mixing strategies. Critical sharpness provides practitioners with a practical tool for diagnosing curvature dynamics and informing data composition choices at scale. More broadly, our work shows that scalable curvature measures can provide actionable insights for large-scale training.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 23 Jan 2026.

Builds on

33 of 46 references.

References · 33

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  6. 2016Progressive Neural Networkscited by 1,361
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  10. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  11. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  12. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
21 more
  1. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  2. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  3. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  4. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  5. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 146
  6. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  7. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 118
  8. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  9. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  10. 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
  11. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  12. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 53
  13. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  14. 2024Why Warmup the Learning Rate? Underlying Mechanisms and Improvementscited by 46
  15. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  16. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  17. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 18
  18. 2024Features are fate: a theory of transfer learning in high-dimensional regressioncited by 13
  19. 2024Gradient descent with generalized Newton's methodcited by 9
  20. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  21. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replaycited by 7
  2. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  3. 2026CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editingcited by 5
  4. 2026Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instabilitycited by 2
  5. 2026On the Optimizer Dependence of Neural Scaling Lawscited by 1
  6. 2026Edge of Stability Selectively Shapes Learning Across the Data Distributioncited by 0
  7. 2026Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adamcited by 0
  8. 2026Muon Sublates the Edge of Stability in LLM Pretrainingcited by 0