mlsanityData to 14 Oct 2026

On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensions

arXiv 2311.03794 ↗First posted 7 Nov 2023math.OC cond-mat.dis-nn stat.MLcited by 21

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

We study the training dynamics of a shallow neural network with quadratic activation functions and quadratic cost in a teacher-student setup. In line with previous works on the same neural architecture, the optimization is performed following the gradient flow on the population risk, where the average over data points is replaced by the expectation over their distribution, assumed to be Gaussian.We first derive convergence properties for the gradient flow and quantify the overparameterization that is necessary to achieve a strong signal recovery. Then, assuming that the teachers and the students at initialization form independent orthonormal families, we derive a high-dimensional limit for the flow and show that the minimal overparameterization is sufficient for strong recovery. We verify by numerical experiments that these results hold for more general initializations.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 23 of its 57 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

23 of 57 references.

References · 23

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  3. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  4. 2014The Loss Surfaces of Multilayer Networkscited by 280
  5. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  6. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  7. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  8. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  9. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  10. 2013Multilayer Networkscited by 87
  11. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 86
  12. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
11 more
  1. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  2. 2023Learning time-scales in two-layers neural networkscited by 57
  3. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  4. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  5. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 43
  6. 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42
  7. 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
  8. 2019Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Modelcited by 25
  9. 2020Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrievalcited by 18
  10. 2019Stationary Points of Shallow Neural Networks with Quadratic Activation Functioncited by 12
  11. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 5.

21 papers.

Sort:

By area:

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  2. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  3. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 20
  4. 2024Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamicscited by 15
  5. 2024Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinationscited by 13
  6. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  7. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 11
  8. 2024Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergencecited by 9
  9. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  10. 2024The Role of the Time-Dependent Hessian in High-Dimensional Optimizationcited by 7
  11. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 6
  12. 2024Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffscited by 3
9 more
  1. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  2. 2025Overparametrization bends the landscape: BBP transitions at initialization in simple Neural Networkscited by 2
  3. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 1
  4. 2026Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrievalcited by 1
  5. 2024Classifying Overlapping Gaussian Mixtures in High Dimensions: From Optimal Classifiers to Neural Netscited by 0
  6. 2025Information-Theoretic Guarantees for Recovering Low-Rank Tensors from Symmetric Rank-One Measurementscited by 0
  7. 2025A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyondcited by 0
  8. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0
  9. 2026How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networks
  2. 2021Learning through atypical "phase transitions" in overparameterized neural networks
  3. 2025Geometry and Optimization of Shallow Polynomial Networks
  4. 2025Convergence Dynamics of Over-Parameterized Score Matching for a Single Gaussian
  5. 2022Flatter, faster: scaling momentum for optimal speedup of SGD
  6. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator
  7. 2025A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Point
  8. 2024The twin peaks of learning neural networks