mlsanityData to 14 Oct 2026

A Loss Curvature Perspective on Training Instability in Deep Learning

arXiv 2110.04369 ↗First posted 8 Oct 2021cs.LG cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

In this work, we study the evolution of the loss Hessian across many classification tasks in order to understand the effect the curvature of the loss has on the training dynamics. Whereas prior work has focused on how different learning rates affect the loss Hessian observed during training, we also analyze the effects of model initialization, architectural choices, and common training heuristics such as gradient clipping and learning rate warmup. Our results demonstrate that successful model and hyperparameter choices allow the early optimization trajectory to either avoid -- or navigate out of -- regions of high curvature and into flatter regions that tolerate a higher learning rate. Our results suggest a unifying perspective on how disparate mitigation strategies for training instability ultimately address the same underlying failure mode of neural network optimization, namely poor conditioning. Inspired by the conditioning perspective, we show that learning rate warmup can improve training stability just as much as batch normalization, layer normalization, MetaInit, GradInit, and Fixup initialization.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

30 of 40 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  8. 2016Wide Residual Networkscited by 3,070
  9. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  12. 2020On Layer Normalization in the Transformer Architecturecited by 922
18 more
  1. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  2. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  3. 2018How Does Batch Normalization Help Optimization?cited by 357
  4. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  5. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  6. 2018An Empirical Model of Large-Batch Trainingcited by 281
  7. 2017Three Factors Influencing Minima in SGDcited by 273
  8. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  9. 2020Understanding the Difficulty of Training Transformerscited by 210
  10. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  11. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  12. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  13. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  14. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
  15. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 88
  16. 2019A Mean Field Theory of Batch Normalizationcited by 58
  17. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  18. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 10; 2024: 15; 2025: 3; 2026: 7.

40 papers.

Sort:

  1. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  2. 2023Understanding plasticity in neural networkscited by 174
  3. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  4. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  5. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  6. 2024Scaling Laws for Precisioncited by 95
  7. 2024Learning with 3D rotations, a hitchhiker's guide to SO(3)cited by 42
  8. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  9. 2022Escaping Saddle Points for Effective Generalization on Class-Imbalanced Datacited by 28
  10. 2024Double Correction Framework for Denoising Recommendationcited by 19
  11. 2024FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher informationcited by 18
  12. 2023Improving Training Stability for Multitask Ranking Models in Recommender Systemscited by 13
28 more
  1. 2022Towards Theoretically Inspired Neural Initialization Optimizationcited by 11
  2. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  3. 2023Data efficiency and extrapolation trends in neural network interatomic potentialscited by 10
  4. 2024Is Flash Attention Stable?cited by 10
  5. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  6. 2024Stable Language Model Pre-training by Reducing Embedding Variabilitycited by 6
  7. 2026The Origin of Edge of Stabilitycited by 6
  8. 2022Optimal learning rate schedules in high-dimensional non-convex optimization problemscited by 4
  9. 2024LTAU-FF: Loss Trajectory Analysis for Uncertainty in Atomistic Force Fieldscited by 4
  10. 2023Confidence-Aware Calibration and Scoring Functions for Curriculum Learningcited by 2
  11. 2024A Parameter Update Balancing Algorithm for Multi-task Ranking Models in Recommendation Systemscited by 2
  12. 2024Torque-Aware Momentumcited by 2
  13. 2025Glocal Smoothness: Line search and adaptive step sizes can help in theory too!cited by 2
  14. 2026Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descentcited by 2
  15. 2023The Disharmony between BN and ReLU Causes Gradient Explosion, but is Offset by the Correlation between Activationscited by 1
  16. 2024Stochastic Subnetwork Annealing: A Regularization Technique for Fine Tuning Pruned Subnetworkscited by 1
  17. 2024Deep Submodular Peripteral Networkscited by 1
  18. 2024Topic-aware Causal Intervention for Counterfactual Detectioncited by 1
  19. 2025CurvaDion: Curvature-Adaptive Distributed Orthonormalizationcited by 1
  20. 2023Robust Reinforcement Learning Objectives for Sequential Recommender Systemscited by 0
  21. 2024Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architecturescited by 0
  22. 2024Advancing Neural Network Performance through Emergence-Promoting Initialization Schemecited by 0
  23. 2025A Machine Learning-Driven Solution for Denoising Inertial Confinement Fusion Imagescited by 0
  24. 2026What do near-optimal learning rate schedules look like?cited by 0
  25. 2026Taming Curvature: Architecture Warm-Up for Stable Transformer Trainingcited by 0
  26. 2026Evaluation-Strategy Gap in Fault Diagnosis of Deep Learning Programscited by 0
  27. 2026Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizonscited by 0
  28. 2026Understanding and Mitigating Training Instability in Large-Scale Recommender Systems: A Cascade Perspectivecited by 0