# On the existence of minimizers in shallow residual ReLU neural network optimization landscapes

arXiv: https://arxiv.org/abs/2302.14690 · HTML: https://mlsanity.com/paper/2302.14690/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 21 of 21)

- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020)
- [Stochastic subgradient method converges on tame functions](https://mlsanity.com/paper/1804.07795/) (2018)
- [Spurious Local Minima are Common in Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/1712.08968/) (2017)
- [Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't](https://mlsanity.com/paper/2009.10713/) (2020)
- [On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems](https://mlsanity.com/paper/2006.11144/) (2020)
- [On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime](https://mlsanity.com/paper/2005.13530/) (2020)
- [Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis](https://mlsanity.com/paper/2105.01650/) (2021)
- [Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes](https://mlsanity.com/paper/2102.09385/) (2021)
- [Local minima in training of neural networks](https://mlsanity.com/paper/1611.06310/) (2016)
- [On minimal representations of shallow ReLU networks](https://mlsanity.com/paper/2108.05643/) (2021)
- [On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks](https://mlsanity.com/paper/2112.09684/) (2021)
- [Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions](https://mlsanity.com/paper/2103.10922/) (2021)
- [Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2108.08106/) (2021)
- [Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks](https://mlsanity.com/paper/2211.15641/) (2022)
- [On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems](https://mlsanity.com/paper/2202.12262/) (2022)
- [Approximation results for Gradient Descent trained Shallow Neural Networks in 1d](https://mlsanity.com/paper/2209.08399/) (2022)
- [Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non-global local minima with high probability](https://mlsanity.com/paper/2212.13111/) (2022)
- [Best k-layer neural network approximations](https://mlsanity.com/paper/1907.01507/) (2019)

## Built on by (most cited first, 6 of 6)

- [Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks](https://mlsanity.com/paper/2402.05155/) (2024)
- [On the existence of optimal shallow feedforward networks with ReLU activation](https://mlsanity.com/paper/2303.03950/) (2023)
- [SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures](https://mlsanity.com/paper/2505.09572/) (2025)
- [PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning](https://mlsanity.com/paper/2505.22085/) (2025)
- [Finite Samples for Shallow Neural Networks](https://mlsanity.com/paper/2503.12744/) (2025)
- [Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks](https://mlsanity.com/paper/2607.02003/) (2026)

## Abstract (arXiv)

In this article, we show existence of minimizers in the loss landscape for residual artificial neural networks (ANNs) with multi-dimensional input layer and one hidden layer with ReLU activation. Our work contrasts earlier results in [D. Gallon, A. Jentzen, and F. Lindner, preprint, arXiv:2211.15641, 2022] and [P. Petersen, M. Raslan, and F. Voigtlaender, Found. Comput. Math., 21 (2021), pp. 375-444] which showed that in many situations minimizers do not exist for common smooth activation functions even in the case where the target functions are polynomials. The proof of the existence property makes use of a closure of the search space containing all functions generated by ANNs and additional discontinuous generalized responses. As we will show, the additional generalized responses in this larger space are suboptimal so that the minimum is attained in the original function class.