mlsanityData to 7 Oct 2026

Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes

arXiv 2204.11326 ↗First posted 24 Apr 2022cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

A quadratic approximation of neural network loss landscapes has been extensively used to study the optimization process of these networks. Though, it usually holds in a very small neighborhood of the minimum, it cannot explain many phenomena observed during the optimization process. In this work, we study the structure of neural network loss functions and its implication on optimization in a region beyond the reach of a good quadratic approximation. Numerically, we observe that neural network loss functions possesses a multiscale structure, manifested in two ways: (1) in a neighborhood of minima, the loss mixes a continuum of scales and grows subquadratically, and (2) in a larger region, the loss shows several separate scales clearly. Using the subquadratic growth, we are able to explain the Edge of Stability phenomenon [5] observed for the gradient descent (GD) method. Using the separate scales, we explain the working mechanism of learning rate decay by simple examples. Finally, we study the origin of the multiscale structure and propose that the non-convexity of the models and the non-uniformity of training data is one of the causes. By constructing a two-layer neural network problem we show that training data with different magnitudes give rise to different scales of the loss function, producing subquadratic growth and multiple separate scales.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 20 of its 23 references that are in the map. First posted on arXiv 24 Apr 2022.

Builds on

20 of 23 references.

References · 20

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  7. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  8. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  9. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  10. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  11. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  12. 2022Understanding the unstable convergence of gradient descentcited by 84
8 more
  1. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  2. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  3. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  4. 2020Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Functioncited by 26
  5. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  6. 2019At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?cited by 16
  7. 2019Loss Patterns of Neural Networkscited by 10
  8. 2018Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descentcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 12; 2024: 11; 2025: 5; 2026: 10.

42 papers.

Sort:

  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  3. 2022Learning threshold neurons via the "edge of stability"cited by 50
  4. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  5. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 34
  6. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  7. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  8. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 20
  9. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  10. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 16
  11. 2023Loss Spike in Training Neural Networkscited by 15
  12. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
30 more
  1. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  2. 2025Hidden Breakthroughs in Language Model Trainingcited by 15
  3. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  4. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  5. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 10
  6. 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 8
  7. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  8. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 6
  9. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  10. 2024From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizescited by 5
  11. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 5
  12. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  13. 2026On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentialscited by 2
  14. 2022A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networkscited by 1
  15. 2024Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignmentcited by 1
  16. 2025Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Lossescited by 1
  17. 2026Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidencecited by 1
  18. 2026Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stabilitycited by 1
  19. 2026Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basincited by 1
  20. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  21. 2023Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization Methodcited by 0
  22. 2024Data-induced multiscale losses and efficient multirate gradient descent schemescited by 0
  23. 2024Corridor Geometry in Gradient-Based Optimizationcited by 0
  24. 2024Deep Model Merging: The Sister of Neural Network Interpretability -- A Surveycited by 0
  25. 2024Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regimecited by 0
  26. 2025Optimization as a Dynamical System: Generative Schedules from Latent ODEscited by 0
  27. 2026A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Modelscited by 0
  28. 2026A Bifurcation Theory Framework for Gradient Descent on the Edge of Stabilitycited by 0
  29. 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0
  30. 2026Beyond Quadratic Loss: The Stability Phase Diagram of Adamcited by 0