mlsanityData to 14 Oct 2026

Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation

arXiv 2108.08106 ↗First posted 18 Aug 2021cs.LG cs.NA math.DScited by 9

Not on the map. Why?

Abstract (arXiv)

The training of artificial neural networks (ANNs) with rectified linear unit (ReLU) activation via gradient descent (GD) type optimization schemes is nowadays a common industrially relevant procedure. Till this day in the scientific literature there is in general no mathematical convergence analysis which explains the numerical success of GD type optimization schemes in the training of ANNs with ReLU activation. GD type optimization schemes can be regarded as temporal discretization methods for the gradient flow (GF) differential equations associated to the considered optimization problem and, in view of this, it seems to be a natural direction of research to first aim to develop a mathematical convergence theory for time-continuous GF differential equations and, thereafter, to aim to extend such a time-continuous convergence theory to implementable time-discrete GD type optimization methods. In this article we establish two basic results for GF differential equations in the training of fully-connected feedforward ANNs with one hidden layer and ReLU activation. In the first main result of this article we establish in the training of such ANNs under the assumption that the probability distribution of the input data of the considered supervised learning problem is absolutely continuous with a bounded density function that every GF differential equation admits for every initial value a solution which is also unique among a suitable class of solutions. In the second main result of this article we prove in the training of such ANNs under the assumption that the target function and the density function of the probability distribution of the input data are piecewise polynomial that every non-divergent GF trajectory converges with an appropriate rate of convergence to a critical point and that the risk of the non-divergent GF trajectory converges with rate 1 to the risk of the critical point.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 22 of its 41 references that are in the map. First posted on arXiv 18 Aug 2021.

Builds on

22 of 41 references.

References · 22

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018On Lazy Training in Differentiable Programmingcited by 675
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  4. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  5. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  7. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 111
  8. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 95
  9. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  10. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  11. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  12. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
10 more
  1. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  2. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 28
  3. 2020A Dynamical Central Limit Theorem for Shallow Neural Networkscited by 23
  4. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  5. 2020Stopping Criteria for, and Strong Convergence of, Stochastic Gradient Descent on Bottou-Curtis-Nocedal Functionscited by 13
  6. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  7. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  8. 2016Unifying abstract inexact convergence theorems and block coordinate variable metric iPianocited by 10
  9. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  10. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 2; 2024: 1; 2025: 0; 2026: 1.

9 papers.

  1. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  2. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  3. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  4. 2023On the existence of minimizers in shallow residual ReLU neural network optimization landscapescited by 6
  5. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6
  6. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  7. 2023On the existence of optimal shallow feedforward networks with ReLU activationcited by 5
  8. 2022On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vectorcited by 0
  9. 2022Normalized gradient flow optimization in the training of ReLU artificial neural networkscited by 0