mlsanityData to 14 Oct 2026

On the Power of Over-parametrization in Neural Networks with Quadratic Activation

arXiv 1803.01206 ↗First posted 3 Mar 2018cs.LG cs.AI math.OCcited by 102

Not on the map. Why?

Abstract (arXiv)

We provide new theoretical insights on why over-parametrization is effective in learning neural networks. For a k hidden node shallow network with quadratic activation and n training data points, we show as long as k ≥ √2n, over-parametrization enables local search algorithms to find a globally optimal solution for general smooth and convex loss functions. Further, despite that the number of parameters may exceed the sample size, using theory of Rademacher complexity, we show with weight decay, the solution also generalizes well if the data is sampled from a regular distribution such as Gaussian. To prove when k≥ √2n, the loss function has benign landscape properties, we adopt an idea from smoothed analysis, which may have other applications in studying loss surfaces of neural networks.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 28; 2022: 19; 2023: 15; 2024: 15; 2025: 17; 2026: 7.

102 papers.

Sort:

By area:

  1. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  2. 2021Tensor Methods in Computer Vision and Deep Learningcited by 52
  3. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  4. 2021Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEscited by 43
  5. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  6. 2022On the influence of over-parameterization in manifold based surrogates and deep neural operatorscited by 29
  7. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  8. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  9. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  10. 2022Interpretable Polynomial Neural Ordinary Differential Equationscited by 24
  11. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  12. 2021Exponentially Many Local Minima in Quantum Neural Networkscited by 22
38 more
  1. 2019On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulatorcited by 21
  2. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  3. 2024The Challenges of the Nonlinear Regime for Physics-Informed Neural Networkscited by 21
  4. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  5. 2021On Expressivity and Trainability of Quadratic Networkscited by 19
  6. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  7. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 19
  8. 2023Demystifying Causal Features on Adversarial Examples and Causal Inoculation for Robust Network by Adversarial Instrumental Variable Regressioncited by 18
  9. 2021Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Timecited by 14
  10. 2021Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networkscited by 14
  11. 2022Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomialscited by 13
  12. 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 12
  13. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  14. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  15. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 11
  16. 2021Local SGD Optimizes Overparameterized Neural Networks in Polynomial Timecited by 10
  17. 2022The Sample Complexity of One-Hidden-Layer Neural Networkscited by 10
  18. 2025Geometry and Optimization of Shallow Polynomial Networkscited by 10
  19. 2022Knowledge Distillation Meets Open-Set Semi-Supervised Learningcited by 9
  20. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  21. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  22. 2021Why Do Local Methods Solve Nonconvex Problems?cited by 8
  23. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  24. 2022Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networkscited by 7
  25. 2025Dynamic Rank Adjustment in Diffusion Policies for Efficient and Flexible Trainingcited by 7
  26. 2021On the Provable Generalization of Recurrent Neural Networkscited by 6
  27. 2024Future Directions in the Theory of Graph Machine Learningcited by 6
  28. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  29. 2024Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Netscited by 6
  30. 2021Estimation of a regression function on a manifold by fully connected deep neural networkscited by 5
  31. 2021Robust Generalization of Quadratic Neural Networks via Function Identificationcited by 5
  32. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
  33. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  34. 2021A Convergence Theory Towards Practical Over-parameterized Deep Neural Networkscited by 4
  35. 2023Initialization-Dependent Sample Complexity of Linear Predictors and Neural Networkscited by 4
  36. 2021Achieving Small Test Error in Mildly Overparameterized Neural Networkscited by 3
  37. 2022Pushing the Efficiency Limit Using Structured Sparse Convolutionscited by 3
  38. 2022COLT: Cyclic Overlapping Lottery Tickets for Faster Pruning of Convolutional Neural Networkscited by 3

50 most cited of 102 All 102 (JSON)