# Symmetry & critical points for a model shallow neural network

arXiv: https://arxiv.org/abs/2003.10576 · HTML: https://mlsanity.com/paper/2003.10576/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2020.

## Built on by (most cited first, 10 of 10)

- [Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II](https://mlsanity.com/paper/2107.10370/) (2021)
- [Symmetry Breaking in Symmetric Tensor Decomposition](https://mlsanity.com/paper/2103.06234/) (2021)
- [Annihilation of Spurious Minima in Two-Layer ReLU Networks](https://mlsanity.com/paper/2210.06088/) (2022)
- [Equivariant bifurcation, quadratic equivariants, and symmetry breaking for the standard representation of Sn](https://mlsanity.com/paper/2107.02422/) (2021)
- [Symmetry & Critical Points for Symmetric Tensor Decomposition Problems](https://mlsanity.com/paper/2306.07886/) (2023)
- [Hidden Minima in Two-Layer ReLU Networks](https://mlsanity.com/paper/2312.16819/) (2023)
- [Symmetry & Critical Points](https://mlsanity.com/paper/2408.14445/) (2024)
- [Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU](https://mlsanity.com/paper/2510.25060/) (2025)
- [Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks](https://mlsanity.com/paper/2109.09612/) (2021)
- [Population loss in shallow ReLU networks: Bias & families of critical points](https://mlsanity.com/paper/2609.30661/) (2026)

## Abstract (arXiv)

We consider the optimization problem associated with fitting two-layer ReLU networks with k hidden neurons, where labels are assumed to be generated by a (teacher) neural network. We leverage the rich symmetry exhibited by such models to identify various families of critical points and express them as power series in k^-frac12. These expressions are then used to derive estimates for several related quantities which imply that not all spurious minima are alike. In particular, we show that while the loss function at certain types of spurious minima decays to zero like k-1, in other cases the loss converges to a strictly positive constant. The methods used depend on symmetry, the geometry of group actions, bifurcation, and Artin's implicit function theorem.