mlsanityData to 7 Oct 2026

Catapult Dynamics and Phase Transitions in Quadratic Nets

arXiv 2301.07737 ↗First posted 18 Jan 2023cs.LG cs.AI stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

Neural networks trained with gradient descent can undergo non-trivial phase transitions as a function of the learning rate. In citelewkowycz2020large it was discovered that wide neural nets can exhibit a catapult phase for super-critical learning rates, where the training loss grows exponentially quickly at early times before rapidly decreasing to a small value. During this phase the top eigenvalue of the neural tangent kernel (NTK) also undergoes significant evolution. In this work, we will prove that the catapult phase exists in a large class of models, including quadratic models and two-layer, homogenous neural nets. To do this, we show that for a certain range of learning rates the weight norm decreases whenever the loss becomes large. We also empirically study learning rates beyond this theoretically derived range and show that the activation map of ReLU nets trained with super-critical learning rates becomes increasingly sparse as we increase the learning rate.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

30 of 36 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2017Deep Neural Networks as Gaussian Processescited by 617
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  11. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  12. 2021The Principles of Deep Learning Theorycited by 169
18 more
  1. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  2. 2014Phase Retrieval via Wirtinger Flow: Theory and Algorithmscited by 123
  3. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  4. 2016A Geometric Analysis of Phase Retrievalcited by 92
  5. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  6. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  7. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  8. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  9. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  10. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  11. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  12. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  13. 2021Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemcited by 23
  14. 2020Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrievalcited by 18
  15. 2017The nonsmooth landscape of phase retrievalcited by 16
  16. 2020Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Widthcited by 11
  17. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  18. 2020Implicit bias of deep linear networks in the large learning rate phasecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 0; 2025: 1; 2026: 1.

8 papers.

  1. 2023Towards provably efficient quantum algorithms for large-scale machine-learning modelscited by 32
  2. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  3. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  4. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  5. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
  6. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 2
  7. 2023Dynamical transition in controllable quantum neural networks with large depthcited by 2
  8. 2026Large Spikes in Stochastic Gradient Descent: A Large-Deviations Viewcited by 0