mlsanityData to 7 Oct 2026

Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations

arXiv 2510.24466 ↗First posted 28 Oct 2025math.OC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The theory of training deep networks has become a central question of modern machine learning and has inspired many practical advancements. In particular, the gradient descent (GD) optimization algorithm has been extensively studied in recent years. A key assumption about GD has appeared in several recent works: the GD map is non-singular -- it preserves sets of measure zero under preimages. Crucially, this assumption has been used to prove that GD avoids saddle points and maxima, and to establish the existence of a computable quantity that determines the convergence to global minima (both for GD and stochastic GD). However, the current literature either assumes the non-singularity of the GD map or imposes restrictive assumptions, such as Lipschitz smoothness of the loss (for example, Lipschitzness does not hold for deep ReLU networks with the cross-entropy loss) and restricts the analysis to GD with small step-sizes. In this paper, we investigate the neural network map as a function on the space of weights and biases. We also prove, for the first time, the non-singularity of the gradient descent (GD) map on the loss landscape of realistic neural network architectures (with fully connected, convolutional, or softmax attention layers) and piecewise analytic activations (which includes sigmoid, ReLU, leaky ReLU, etc.) for almost all step-sizes. Our work significantly extends the existing results on the convergence of GD and SGD by guaranteeing that they apply to practical neural network settings and has the potential to unlock further exploration of learning dynamics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

22 of 35 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  3. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  4. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  5. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  6. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  7. 2022Understanding the unstable convergence of gradient descentcited by 84
  8. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  9. 2018Mad Max: Affine Spline Insights into Deep Learningcited by 54
  10. 2019The Geometry of Deep Networks: Power Diagram Subdivisioncited by 53
  11. 2019First-order methods almost always avoid saddle points: the case of vanishing step-sizescited by 35
  12. 2016Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regionscited by 33
10 more
  1. 2023SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundariescited by 31
  2. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 25
  3. 2014On the saddle point problem for non-convex optimizationcited by 19
  4. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  5. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  6. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  7. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
  8. 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 7
  9. 2020The critical locus of overparameterized neural networkscited by 6
  10. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5

Built on by

Not cited yet.