# Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry

arXiv: https://arxiv.org/abs/2008.01805 · HTML: https://mlsanity.com/paper/2008.01805/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2020.

## Built on by (most cited first, 17 of 17)

- [AdaGrad under Anisotropic Smoothness](https://mlsanity.com/paper/2406.15244/) (2024)
- [On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias](https://mlsanity.com/paper/2205.09072/) (2022)
- [Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II](https://mlsanity.com/paper/2107.10370/) (2021)
- [Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums](https://mlsanity.com/paper/2110.14109/) (2021)
- [Symmetry Breaking in Symmetric Tensor Decomposition](https://mlsanity.com/paper/2103.06234/) (2021)
- [Annihilation of Spurious Minima in Two-Layer ReLU Networks](https://mlsanity.com/paper/2210.06088/) (2022)
- [Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise](https://mlsanity.com/paper/2312.14567/) (2023)
- [A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models](https://mlsanity.com/paper/2506.02269/) (2025)
- [Equivariant bifurcation, quadratic equivariants, and symmetry breaking for the standard representation of Sn](https://mlsanity.com/paper/2107.02422/) (2021)
- [A Classification of G-invariant Shallow Neural Networks](https://mlsanity.com/paper/2205.09219/) (2022)
- [Symmetry & Critical Points for Symmetric Tensor Decomposition Problems](https://mlsanity.com/paper/2306.07886/) (2023)
- [Hidden Minima in Two-Layer ReLU Networks](https://mlsanity.com/paper/2312.16819/) (2023)
- [Symmetry & Critical Points](https://mlsanity.com/paper/2408.14445/) (2024)
- [Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU](https://mlsanity.com/paper/2510.25060/) (2025)
- [A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Point](https://mlsanity.com/paper/2512.15606/) (2025)
- [An Equivariance Toolbox for Learning Dynamics](https://mlsanity.com/paper/2512.21447/) (2025)
- [Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://mlsanity.com/paper/2607.07845/) (2026)

## Abstract (arXiv)

We consider the optimization problem associated with fitting two-layers ReLU networks with respect to the squared loss, where labels are generated by a target network. We leverage the rich symmetry structure to analytically characterize the Hessian at various families of spurious minima in the natural regime where the number of inputs d and the number of hidden neurons k is finite. In particular, we prove that for d≥ k standard Gaussian inputs: (a) of the dk eigenvalues of the Hessian, dk - O(d) concentrate near zero, (b) Omega(d) of the eigenvalues grow linearly with k. Although this phenomenon of extremely skewed spectrum has been observed many times before, to our knowledge, this is the first time it has been established rigorously. Our analytic approach uses techniques, new to the field, from symmetry breaking and representation theory, and carries important implications for our ability to argue about statistical generalization through local curvature.