mlsanityData to 14 Oct 2026

SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures

arXiv 2505.09572 ↗First posted 14 May 2025cs.LG math.LO math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

We study gradient flows for loss landscapes of fully connected feedforward neural networks with commonly used continuously differentiable activation functions such as the logistic, hyperbolic tangent, softplus or GELU function. We prove that the gradient flow either converges to a critical point or diverges to infinity while the loss converges to an asymptotic critical value. Moreover, we prove the existence of a threshold varepsilon>0 such that the loss value of any gradient flow initialized at most varepsilon above the optimal level converges to it. For polynomial target functions and sufficiently big architecture and data set, we prove that the optimal loss value is zero and can only be realized asymptotically. From this setting, we deduce our main result that any gradient flow with sufficiently good initialization diverges to infinity. Our proof heavily relies on the geometry of o-minimal structures. We confirm these theoretical findings with numerical experiments and extend our investigation to more realistic scenarios, where we observe an analogous behavior.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 38 of its 84 references that are in the map. First posted on arXiv 14 May 2025.

Builds on

38 of 84 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  7. 2016Error bounds for approximations with deep ReLU networkscited by 574
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  9. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  10. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  11. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 179
  12. 2020Directional convergence and alignment in deep learningcited by 178
26 more
  1. 2018Stochastic subgradient method converges on tame functionscited by 124
  2. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 91
  3. 2021On the approximation of functions by tanh neural networkscited by 81
  4. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  5. 2018Solving the Kolmogorov PDE by means of deep learningcited by 41
  6. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  7. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 28
  8. 2020Convergence of constant step stochastic gradient descent for non-smooth non-convex functionscited by 27
  9. 2024Convergence rates for the Adam optimizercited by 15
  10. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  11. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 11
  12. 2022Blow up phenomena for gradient descent optimization methods in the training of artificial neural networkscited by 9
  13. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  14. 2024Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activationcited by 8
  15. 2022Approximation results for Gradient Descent trained Shallow Neural Networks in 1dcited by 7
  16. 2022Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non-global local minima with high probabilitycited by 7
  17. 2019Analysis of Deep Neural Networks with Quasi-optimal polynomial approximation ratescited by 6
  18. 2023On the existence of minimizers in shallow residual ReLU neural network optimization landscapescited by 6
  19. 2023Global stability of first-order methods for coercive tame functionscited by 6
  20. 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
  21. 2023Global convergence of the gradient method for functions definable in o-minimal structurescited by 5
  22. 2023On the existence of optimal shallow feedforward networks with ReLU activationcited by 5
  23. 2024Learning-rate-free Momentum SGD with Reshuffling Converges in Nonsmooth Nonconvex Optimizationcited by 4
  24. 2021A convex function satisfying the Lojasiewicz inequality but failing the gradient conjecture both at zero and infinitycited by 3
  25. 2023Certifying the absence of spurious local minima at infinitycited by 2
  26. 2024Stability of first-order methods in tame optimizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Deep Learning as the Disciplined Construction of Tame Objectscited by 3
  2. 2025PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learningcited by 1
  3. 2026Central limit theorem for the averaged Adam optimizercited by 1
  4. 2025Exact Subgraph Isomorphism Network with Mixed L0,2 Norm Constraint for Predictive Graph Miningcited by 0
  5. 2025Nonconvex Penalized LAD Estimation in Partial Linear Models with DNNs: Asymptotic Analysis and Proximal Algorithmscited by 0