# Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation

arXiv: https://arxiv.org/abs/2410.10533 · HTML: https://mlsanity.com/paper/2410.10533/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2024.

## Built on by (most cited first, 8 of 8)

- [Sharp higher order convergence rates for the Adam optimizer](https://mlsanity.com/paper/2504.19426/) (2025)
- [Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks](https://mlsanity.com/paper/2503.01660/) (2025)
- [SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures](https://mlsanity.com/paper/2505.09572/) (2025)
- [Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method](https://mlsanity.com/paper/2603.18899/) (2026)
- [Effective continuous equations for adaptive SGD: a stochastic analysis view](https://mlsanity.com/paper/2509.21614/) (2025)
- [On the Convergence of Adam, Revisited](https://mlsanity.com/paper/2607.03519/) (2026)
- [Error analysis for the deep Kolmogorov method](https://mlsanity.com/paper/2508.17167/) (2025)
- [Convergence rates for the RMSprop optimizer with full control of the hyperparameters](https://mlsanity.com/paper/2608.30382/) (2026)

## Abstract (arXiv)

Deep learning methods - consisting of a class of deep neural networks (DNNs) trained by a stochastic gradient descent (SGD) optimization method - are nowadays key tools to solve data driven supervised learning problems. Despite the great success of SGD methods in the training of DNNs, it remains a fundamental open problem of research to explain the success and the limitations of such methods in rigorous theoretical terms. In particular, even in the standard setup of data driven supervised learning problems, it remained an open research problem to prove (or disprove) that SGD methods converge in the training of DNNs with the popular rectified linear unit (ReLU) activation function with high probability to global minimizers in the optimization landscape. In this work we answer this question negatively. Specifically, in this work we prove for a large class of SGD methods that the considered optimizer does with high probability not converge to global minimizers of the optimization problem. It turns out that the probability to not converge to a global minimizer converges at least exponentially quickly to one as the width of the first hidden layer of the ANN and the depth of the ANN, respectively, increase. The general non-convergence results of this work do not only apply to the plain vanilla standard SGD method but also to a large class of accelerated and adaptive SGD methods such as the momentum SGD, the Nesterov accelerated SGD, the Adagrad, the RMSProp, the Adam, the Adamax, the AMSGrad, and the Nadam optimizers.