# Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks

arXiv: https://arxiv.org/abs/2405.17501 · HTML: https://mlsanity.com/paper/2405.17501/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 26 May 2024.

## Builds on (most cited first, 12 of 12)

- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [The loss surface of deep and wide neural networks](https://mlsanity.com/paper/1704.08045/) (2017)
- [The Global Landscape of Neural Networks: An Overview](https://mlsanity.com/paper/2007.01429/) (2020)
- [Understanding Generalization through Visualizations](https://mlsanity.com/paper/1906.03291/) (2019)
- [On Connected Sublevel Sets in Deep Learning](https://mlsanity.com/paper/1901.07417/) (2019)
- [Semi-flat minima and saddle points by embedding neural networks to overparameterization](https://mlsanity.com/paper/1906.04868/) (2019)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [On the Benefit of Width for Neural Networks: Disappearance of Bad Basins](https://mlsanity.com/paper/1812.11039/) (2018)
- [Loss Patterns of Neural Networks](https://mlsanity.com/paper/1910.03867/) (2019)
- [Optimistic Estimate Uncovers the Potential of Nonlinear Models](https://mlsanity.com/paper/2307.08921/) (2023)
- [Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization](https://mlsanity.com/paper/2309.00508/) (2023)

## Built on by (most cited first, 2 of 2)

- [Linear Independence of Generalized Neurons and Related Functions](https://mlsanity.com/paper/2410.03693/) (2024)
- [A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks](https://mlsanity.com/paper/2606.04327/) (2026)

## Abstract (arXiv)

This paper presents a comprehensive analysis of critical point sets in two-layer neural networks. To study such complex entities, we introduce the critical embedding operator and critical reduction operator as our tools. Given a critical point, we use these operators to uncover the whole underlying critical set representing the same output function, which exhibits a hierarchical structure. Furthermore, we prove existence of saddle branches for any critical set whose output function can be represented by a narrower network. Our results provide a solid foundation to the further study of optimization and training behavior of neural networks.