# On the existence of optimal shallow feedforward networks with ReLU activation

arXiv: https://arxiv.org/abs/2303.03950 · HTML: https://mlsanity.com/paper/2303.03950/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 12 of 12)

- [Stochastic subgradient method converges on tame functions](https://mlsanity.com/paper/1804.07795/) (2018)
- [Spurious Local Minima are Common in Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/1712.08968/) (2017)
- [Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't](https://mlsanity.com/paper/2009.10713/) (2020)
- [On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems](https://mlsanity.com/paper/2006.11144/) (2020)
- [Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes](https://mlsanity.com/paper/2102.09385/) (2021)
- [Local minima in training of neural networks](https://mlsanity.com/paper/1611.06310/) (2016)
- [On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks](https://mlsanity.com/paper/2112.09684/) (2021)
- [Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2108.08106/) (2021)
- [Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks](https://mlsanity.com/paper/2211.15641/) (2022)
- [On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems](https://mlsanity.com/paper/2202.12262/) (2022)
- [On the existence of minimizers in shallow residual ReLU neural network optimization landscapes](https://mlsanity.com/paper/2302.14690/) (2023)
- [Best k-layer neural network approximations](https://mlsanity.com/paper/1907.01507/) (2019)

## Built on by (most cited first, 5 of 5)

- [Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks](https://mlsanity.com/paper/2402.05155/) (2024)
- [Most ReLU Networks Admit Identifiable Parameters](https://mlsanity.com/paper/2605.03601/) (2026)
- [SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures](https://mlsanity.com/paper/2505.09572/) (2025)
- [The Symmetries of Three-Layer ReLU Networks](https://mlsanity.com/paper/2605.18319/) (2026)
- [PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning](https://mlsanity.com/paper/2505.22085/) (2025)

## Abstract (arXiv)

We prove existence of global minima in the loss landscape for the approximation of continuous target functions using shallow feedforward artificial neural networks with ReLU activation. This property is one of the fundamental artifacts separating ReLU from other commonly used activation functions. We propose a kind of closure of the search space so that in the extended space minimizers exist. In a second step, we show under mild assumptions that the newly added functions in the extension perform worse than appropriate representable ReLU networks. This then implies that the optimal response in the extended target space is indeed the response of a ReLU network.