# Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU

arXiv: https://arxiv.org/abs/2510.25060 · HTML: https://mlsanity.com/paper/2510.25060/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 11 of 11)

- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Theoretical insights into the optimization landscape of over-parameterized shallow neural networks](https://mlsanity.com/paper/1707.04926/) (2017)
- [Spurious Local Minima are Common in Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/1712.08968/) (2017)
- [The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks](https://mlsanity.com/paper/2006.01005/) (2020)
- [Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry](https://mlsanity.com/paper/2008.01805/) (2020)
- [On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting](https://mlsanity.com/paper/2106.06251/) (2021)
- [Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II](https://mlsanity.com/paper/2107.10370/) (2021)
- [Symmetry & critical points for a model shallow neural network](https://mlsanity.com/paper/2003.10576/) (2020)
- [Symmetry Breaking in Symmetric Tensor Decomposition](https://mlsanity.com/paper/2103.06234/) (2021)
- [Geometry and Optimization of Shallow Polynomial Networks](https://mlsanity.com/paper/2501.06074/) (2025)
- [Annihilation of Spurious Minima in Two-Layer ReLU Networks](https://mlsanity.com/paper/2210.06088/) (2022)

## Built on by (most cited first, 1 of 1)

- [Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking](https://mlsanity.com/paper/2610.09919/) (2026)

## Abstract (arXiv)

In this work, we study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation. Under Gaussian inputs and equal layer width k, (1) we establish, based on the equivariant gradient degree, a theoretical framework, applicable to any number of neurons k>= 4, to detect bifurcation of critical points with associated symmetries from global minimum as leaky parameter α varies. Typically, our analysis reveals that a multi-mode degeneracy consistently occurs at the critical number 0, independent of k. (2) As a by-product, we further show that such bifurcations are width-independent, arise only for nonnegative α and that the global minimum undergoes no further symmetry-breaking instability throughout the engineering regime α in range (0,1). An explicit example with k=5 is presented to illustrate the framework and exhibit the resulting bifurcation together with their symmetries.