mlsanityData to 7 Oct 2026

Quadratic models for understanding catapult dynamics of neural networks

arXiv 2205.11787 ↗First posted 24 May 2022cs.LG math.OC stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

While neural networks can be approximated by linear models as their width increases, certain properties of wide neural networks cannot be captured by linear models. In this work we show that recently proposed Neural Quadratic Models can exhibit the "catapult phase" [Lewkowycz et al. 2020] that arises when training such models with large learning rates. We then empirically show that the behaviour of neural quadratic models parallels that of neural networks in generalization, especially in the catapult phase regime. Our analysis further demonstrates that quadratic models can be an effective tool for analysis of neural networks.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

21 of 29 references.

References · 21

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  7. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  8. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  9. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  10. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  11. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  12. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
9 more
  1. 2019How do infinite width bounded norm networks look in function space?cited by 105
  2. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  3. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  4. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  5. 2021What can linearized neural networks actually say about generalization?cited by 49
  6. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  7. 2021Properties of the After Kernelcited by 27
  8. 2018Lyapunov Theory for Discrete Time Systemscited by 15
  9. 2020Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 3; 2025: 5; 2026: 1.

17 papers.

Sort:

  1. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  2. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  3. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 24
  4. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  5. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  6. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 15
  7. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 15
  8. 2024FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneitycited by 13
  9. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 9
  10. 2023Benign Oscillation of Stochastic Gradient Descent with Large Learning Ratescited by 8
  11. 2025Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Modelscited by 7
  12. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
5 more
  1. 2025Learning a Single Index Model from Anisotropic Data with vanilla Stochastic Gradient Descentcited by 4
  2. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 2
  3. 2025A dynamic view of some anomalous phenomena in SGDcited by 2
  4. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  5. 2026Large Spikes in Stochastic Gradient Descent: A Large-Deviations Viewcited by 0