mlsanityData to 14 Oct 2026

Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

arXiv 2006.15459 ↗First posted 27 Jun 2020cs.LG stat.MLcited by 50

Not on the map. Why?

Abstract (arXiv)

We study the dynamics of optimization and the generalization properties of one-hidden layer neural networks with quadratic activation function in the over-parametrized regime where the layer width m is larger than the input dimension d. We consider a teacher-student scenario where the teacher has the same structure as the student with a hidden layer of smaller width m^*≤ m. We describe how the empirical loss landscape is affected by the number n of data samples and the width m^* of the teacher network. In particular we determine how the probability that there be no spurious minima on the empirical loss depends on n, d, and m^*, thereby establishing conditions under which the neural network can in principle recover the teacher. We also show that under the same conditions gradient descent dynamics on the empirical loss converges and leads to small generalization error, i.e. it enables recovery in practice. Finally we characterize the time-convergence rate of gradient descent in the limit of a large number of samples. These results are confirmed by numerical experiments.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 4; 2023: 11; 2024: 2; 2025: 15; 2026: 8.

50 papers.

Sort:

By area:

  1. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  2. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  3. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  4. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  5. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  6. 2023Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspectivecited by 26
  7. 2021Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemcited by 23
  8. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  9. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  10. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 20
  11. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  12. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
38 more
  1. 2021On Expressivity and Trainability of Quadratic Networkscited by 19
  2. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  3. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  4. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 15
  5. 2021Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Timecited by 14
  6. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 14
  7. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 13
  8. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  9. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  10. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 11
  11. 2025Geometry and Optimization of Shallow Polynomial Networkscited by 10
  12. 2024Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networkscited by 9
  13. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  14. 2023Catapult Dynamics and Phase Transitions in Quadratic Netscited by 8
  15. 2023Geometric structure of Deep Learning networks and construction of global mathcal L2 minimizerscited by 6
  16. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 6
  17. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  18. 2023Embeddings between Barron spaces with higher order activation functionscited by 4
  19. 2023Geometric structure of shallow neural networks and constructive mathcal L2 cost minimizationcited by 4
  20. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  21. 2025Linearly Separable Features in Shallow Nonlinear Networks: Width Scales Polynomially with Intrinsic Data Dimensioncited by 2
  22. 2025Overparametrization bends the landscape: BBP transitions at initialization in simple Neural Networkscited by 2
  23. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  24. 2022SQ-Swin: a Pretrained Siamese Quadratic Swin Transformer for Lettuce Browning Predictioncited by 1
  25. 2023The Local Landscape of Phase Retrieval Under Limited Samplescited by 1
  26. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 1
  27. 2025Curse of Dimensionality in Neural Network Optimizationcited by 1
  28. 2026Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrievalcited by 1
  29. 2026Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networkscited by 1
  30. 2021Neural Networks with Complex-Valued Weights Have No Spurious Local Minimacited by 0
  31. 2023On non-approximability of zero loss global mathcal L2 minimizers by gradient descent in Deep Learningcited by 0
  32. 2025Information-Theoretic Guarantees for Recovering Low-Rank Tensors from Symmetric Rank-One Measurementscited by 0
  33. 2025Enhancing Expressivity of Quantum Neural Networks Based on the SWAP testcited by 0
  34. 2025I2I-PR: Deep Iterative Refinement for Phase Retrieval using Image-to-Image Diffusion Modelscited by 0
  35. 2025A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyondcited by 0
  36. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0
  37. 2026Why ReLU? A Bit-Model Dichotomy for Deep Network Trainingcited by 0
  38. 2026Teacher Geometry Shapes Learnability in Teacher-Student Networkscited by 0