mlsanityData to 7 Oct 2026

On the existence of minimizers in shallow residual ReLU neural network optimization landscapes

arXiv 2302.14690 ↗First posted 28 Feb 2023math.OC cs.LG cs.NAcited by 6

Not on the map. Why?

Abstract (arXiv)

In this article, we show existence of minimizers in the loss landscape for residual artificial neural networks (ANNs) with multi-dimensional input layer and one hidden layer with ReLU activation. Our work contrasts earlier results in [D. Gallon, A. Jentzen, and F. Lindner, preprint, arXiv:2211.15641, 2022] and [P. Petersen, M. Raslan, and F. Voigtlaender, Found. Comput. Math., 21 (2021), pp. 375-444] which showed that in many situations minimizers do not exist for common smooth activation functions even in the case where the target functions are polynomials. The proof of the existence property makes use of a closure of the search space containing all functions generated by ANNs and additional discontinuous generalized responses. As we will show, the additional generalized responses in this larger space are suboptimal so that the minimum is attained in the original function class.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

21 of 38 references.

References · 21

  1. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  2. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  3. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  4. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  5. 2018Stochastic subgradient method converges on tame functionscited by 120
  6. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  7. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
  8. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 79
  9. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  10. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  11. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 27
  12. 2016Local minima in training of neural networkscited by 19
9 more
  1. 2021On minimal representations of shallow ReLU networkscited by 13
  2. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  3. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  4. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  5. 2022Blow up phenomena for gradient descent optimization methods in the training of artificial neural networkscited by 9
  6. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8
  7. 2022Approximation results for Gradient Descent trained Shallow Neural Networks in 1dcited by 7
  8. 2022Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non-global local minima with high probabilitycited by 7
  9. 2019Best k-layer neural network approximationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 3; 2026: 1.

6 papers.

  1. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  2. 2023On the existence of optimal shallow feedforward networks with ReLU activationcited by 5
  3. 2025SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structurescited by 5
  4. 2025PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learningcited by 1
  5. 2025Finite Samples for Shallow Neural Networkscited by 0
  6. 2026Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networkscited by 0