# Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non-global local minima with high probability

arXiv: https://arxiv.org/abs/2212.13111 · HTML: https://mlsanity.com/paper/2212.13111/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Dec 2022.

## Built on by (most cited first, 7 of 7)

- [Convergence rates for the Adam optimizer](https://mlsanity.com/paper/2407.21078/) (2024)
- [Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks](https://mlsanity.com/paper/2402.05155/) (2024)
- [On the existence of minimizers in shallow residual ReLU neural network optimization landscapes](https://mlsanity.com/paper/2302.14690/) (2023)
- [SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures](https://mlsanity.com/paper/2505.09572/) (2025)
- [A brief review of the Deep BSDE method for solving high-dimensional partial differential equations](https://mlsanity.com/paper/2505.17032/) (2025)
- [Approximation Results for Gradient Descent trained Neural Networks](https://mlsanity.com/paper/2309.04860/) (2023)
- [Approximation and Gradient Descent Training with Neural Networks](https://mlsanity.com/paper/2405.11696/) (2024)

## Abstract (arXiv)

Gradient descent (GD) methods for the training of artificial neural networks (ANNs) belong nowadays to the most heavily employed computational schemes in the digital world. Despite the compelling success of such methods, it remains an open problem to provide a rigorous theoretical justification for the success of GD methods in the training of ANNs. The main difficulty is that the optimization risk landscapes associated to ANNs usually admit many non-optimal critical points (saddle points as well as non-global local minima) whose risk values are strictly larger than the optimal risk value. It is a key contribution of this article to overcome this obstacle in certain simplified shallow ANN training situations. In such simplified ANN training scenarios we prove that the gradient flow (GF) dynamics with only one random initialization overcomes with high probability all bad non-global local minima (all non-global local minima whose risk values are much larger than the risk value of the global minima) and converges with high probability to a good critical point (a critical point whose risk value is very close to the optimal risk value of the global minima). This analysis allows us to establish convergence in probability to zero of the risk value of the GF trajectories with convergence rates as the ANN training time and the width of the ANN increase to infinity. We complement the analytical findings of this work with extensive numerical simulations for shallow and deep ANNs: All these numerical simulations strongly suggest that with high probability the considered GD method (stochastic GD or Adam) overcomes all bad non-global local minima, does not converge to a global minimum, but does converge to a good non-optimal critical point whose risk value is very close to the optimal risk value.