mlsanityData to 14 Oct 2026

On the existence of optimal shallow feedforward networks with ReLU activation

arXiv 2303.03950 ↗First posted 6 Mar 2023cs.LG cs.NA math.NAcited by 5

Not on the map. Why?

Abstract (arXiv)

We prove existence of global minima in the loss landscape for the approximation of continuous target functions using shallow feedforward artificial neural networks with ReLU activation. This property is one of the fundamental artifacts separating ReLU from other commonly used activation functions. We propose a kind of closure of the search space so that in the extended space minimizers exist. In a second step, we show under mild assumptions that the newly added functions in the extension perform worse than appropriate representable ReLU networks. This then implies that the optimal response in the extended target space is indeed the response of a ReLU network.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 12 of its 27 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

12 of 27 references.

References · 12

  1. 2018Stochastic subgradient method converges on tame functionscited by 124
  2. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  3. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 95
  4. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
  5. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 28
  6. 2016Local minima in training of neural networkscited by 19
  7. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  8. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  9. 2022Blow up phenomena for gradient descent optimization methods in the training of artificial neural networkscited by 9
  10. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8
  11. 2023On the existence of minimizers in shallow residual ReLU neural network optimization landscapescited by 6
  12. 2019Best k-layer neural network approximationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  2. 2026Most ReLU Networks Admit Identifiable Parameterscited by 6
  3. 2025SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structurescited by 5
  4. 2026The Symmetries of Three-Layer ReLU Networkscited by 3
  5. 2025PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learningcited by 1