mlsanityData to 14 Oct 2026

Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation

arXiv 2107.04479 ↗First posted 9 Jul 2021cs.LG cs.NA math.DScited by 11

Not on the map. Why?

Abstract (arXiv)

Gradient descent (GD) type optimization schemes are the standard methods to train artificial neural networks (ANNs) with rectified linear unit (ReLU) activation. Such schemes can be considered as discretizations of gradient flows (GFs) associated to the training of ANNs with ReLU activation and most of the key difficulties in the mathematical convergence analysis of GD type optimization schemes in the training of ANNs with ReLU activation seem to be already present in the dynamics of the corresponding GF differential equations. It is the key subject of this work to analyze such GF differential equations in the training of ANNs with ReLU activation and three layers (one input layer, one hidden layer, and one output layer). In particular, in this article we prove in the case where the target function is possibly multi-dimensional and continuous and in the case where the probability distribution of the input data is absolutely continuous with respect to the Lebesgue measure that the risk of every bounded GF trajectory converges to the risk of a critical point. In addition, in this article we show in the case of a 1-dimensional affine linear target function and in the case where the probability distribution of the input data coincides with the standard uniform distribution that the risk of every bounded GF trajectory converges to zero if the initial risk is sufficiently small. Finally, in the special situation where there is only one neuron on the hidden layer (1-dimensional hidden layer) we strengthen the above named result for affine linear target functions by proving that that the risk of every (not necessarily bounded) GF trajectory converges to zero if the initial risk is sufficiently small.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

18 of 27 references.

References · 18

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018On Lazy Training in Differentiable Programmingcited by 675
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  5. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  6. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 95
  7. 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 72
  8. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  9. 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
  10. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  11. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  12. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
6 more
  1. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  2. 2018Solving the Kolmogorov PDE by means of deep learningcited by 41
  3. 2020A Dynamical Central Limit Theorem for Shallow Neural Networkscited by 23
  4. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  5. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  6. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 1; 2023: 2; 2024: 2; 2025: 2; 2026: 1.

11 papers.

  1. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  2. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  3. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  4. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  5. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  6. 2023Operator theory, kernels, and Feedforward Neural Networkscited by 2
  7. 2025Convergence of Shallow ReLU Networks on Weakly Interacting Datacited by 2
  8. 2026Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimizationcited by 1
  9. 2022Normalized gradient flow optimization in the training of ReLU artificial neural networkscited by 0
  10. 2024Predicting the fatigue life of asphalt concrete using neural networkscited by 0
  11. 2025Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activationscited by 0