mlsanityData to 14 Oct 2026

Gradient Flow Convergence Guarantee for General Neural Network Architectures

arXiv 2509.23887 ↗First posted 28 Sep 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

A key challenge in modern deep learning theory is to explain the remarkable success of gradient-based optimization methods when training large-scale, complex deep neural networks. Though linear convergence of such methods has been proved for a handful of specific architectures, a united theory still evades researchers. This article presents a unified proof for linear convergence of continuous gradient descent, also called gradient flow, while training any neural network with piecewise non-zero polynomial activations or ReLU, sigmoid activations. Our primary contribution is a single, general theorem that not only covers architectures for which this result was previously unknown but also consolidates existing results under weaker assumptions. While our focus is theoretical and our results are only exact in the infinitesimal step size limit, we nevertheless find excellent empirical agreement between the predictions of our result and those of the practical step-size gradient descent method.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

31 of 43 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  7. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  9. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 881
  10. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  11. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  12. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
19 more
  1. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  2. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  3. 2017How to Escape Saddle Points Efficientlycited by 285
  4. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  5. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  6. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  7. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  8. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  9. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  10. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  11. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  12. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  13. 2023Scalable Optimal Transport Methods in Machine Learning: A Contemporary Surveycited by 27
  14. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  15. 2022Convergence of gradient descent for deep neural networkscited by 24
  16. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 13
  17. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  18. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  19. 2025A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Modelscited by 4

Built on by

Not cited yet.