# Critical Points of Neural Networks: Analytical Forms and Landscape Properties

arXiv: https://arxiv.org/abs/1710.11205 · HTML: https://mlsanity.com/paper/1710.11205/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2017.

## Built on by (most cited first, 7 of 7)

- [When Expressivity Meets Trainability: Fewer than n Neurons Can Work](https://mlsanity.com/paper/2210.12001/) (2022)
- [Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/2102.13233/) (2021)
- [Understanding Deep Learning via Decision Boundary](https://mlsanity.com/paper/2206.01515/) (2022)
- [The Landscape of Multi-Layer Linear Neural Network From the Perspective of Algebraic Geometry](https://mlsanity.com/paper/2102.04338/) (2021)
- [Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensions](https://mlsanity.com/paper/2206.03254/) (2022)
- [Impact of Bottleneck Layers and Skip Connections on the Generalization of Linear Denoising Autoencoders](https://mlsanity.com/paper/2505.24668/) (2025)
- [Benign Loss Landscapes Can Coexist with Worst-Case Hardness](https://mlsanity.com/paper/2609.13057/) (2026)

## Abstract (arXiv)

Due to the success of deep learning to solving a variety of challenging machine learning tasks, there is a rising interest in understanding loss functions for training neural networks from a theoretical aspect. Particularly, the properties of critical points and the landscape around them are of importance to determine the convergence performance of optimization algorithms. In this paper, we provide full (necessary and sufficient) characterization of the analytical forms for the critical points (as well as global minimizers) of the square loss functions for various neural networks. We show that the analytical forms of the critical points characterize the values of the corresponding loss functions as well as the necessary and sufficient conditions to achieve global minimum. Furthermore, we exploit the analytical forms of the critical points to characterize the landscape properties for the loss functions of these neural networks. One particular conclusion is that: The loss function of linear networks has no spurious local minimum, while the loss function of one-hidden-layer nonlinear networks with ReLU activation function does have local minimum that is not global minimum.