# Stationary Points of Shallow Neural Networks with Quadratic Activation Function

arXiv: https://arxiv.org/abs/1912.01599 · HTML: https://mlsanity.com/paper/1912.01599/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2019.

## Built on by (most cited first, 12 of 12)

- [Grokking as a First Order Phase Transition in Two Layer Networks](https://mlsanity.com/paper/2310.03789/) (2023)
- [Bayes-optimal learning of an extensive-width neural network from quadratically many samples](https://mlsanity.com/paper/2408.03733/) (2024)
- [Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks](https://mlsanity.com/paper/2109.09304/) (2021)
- [On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensions](https://mlsanity.com/paper/2311.03794/) (2023)
- [Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time](https://mlsanity.com/paper/2101.02429/) (2021)
- [Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinations](https://mlsanity.com/paper/2406.11828/) (2024)
- [The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks](https://mlsanity.com/paper/2505.17958/) (2025)
- [The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks](https://mlsanity.com/paper/2602.16340/) (2026)
- [Greedy Pruning with Group Lasso Provably Generalizes for Matrix Sensing](https://mlsanity.com/paper/2303.11453/) (2023)
- [Neural Networks with Complex-Valued Weights Have No Spurious Local Minima](https://mlsanity.com/paper/2103.07287/) (2021)
- [A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyond](https://mlsanity.com/paper/2510.19382/) (2025)
- [How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks](https://mlsanity.com/paper/2606.28242/) (2026)

## Abstract (arXiv)

We consider the teacher-student setting of learning shallow neural networks with quadratic activations and planted weight matrix W^*∈ℝm× d, where m is the width of the hidden layer and d≤ m is the data dimension. We study the optimization landscape associated with the empirical and the population squared risk of the problem. Under the assumption the planted weights are full-rank we obtain the following results. First, we establish that the landscape of the empirical risk admits an "energy barrier" separating rank-deficient W from W^*: if W is rank deficient, then its risk is bounded away from zero by an amount we quantify. We then couple this result by showing that, assuming number N of samples grows at least like a polynomial function of d, all full-rank approximate stationary points of the empirical risk are nearly global optimum. These two results allow us to prove that gradient descent, when initialized below the energy barrier, approximately minimizes the empirical risk and recovers the planted weights in polynomial-time. Next, we show that initializing below this barrier is in fact easily achieved when the weights are randomly generated under relatively weak assumptions. We show that provided the network is sufficiently overparametrized, initializing with an appropriate multiple of the identity suffices to obtain a risk below the energy barrier. At a technical level, the last result is a consequence of the semicircle law for the Wishart ensemble and could be of independent interest. Finally, we study the minimizers of the empirical risk and identify a simple necessary and sufficient geometric condition on the training data under which any minimizer has necessarily zero generalization error. We show that as soon as N≥ N^*=d(d+1)/2, randomly generated data enjoys this geometric condition almost surely, while that ceases to be true if N<N^*.