A convergence result of a continuous model of deep learning via a Lojasiewicz--Simon inequality
Not on the map. Why?
Abstract (arXiv)
We study an idealized training process for deep neural networks in a continuous-depth, mean-field model in which each layer is parameterized by a probability measure on a Euclidean parameter space. The training dynamics are formulated as a Wasserstein-type gradient flow of an objective with a fixed L2-regularization. Under suitable analyticity and growth assumptions, together with a coercivity assumption and sufficient regularity of the initial data, we prove that every curve of maximal slope converges to a single critical point of the objective as the training time tends to infinity. The proof combines compactness of the curve with a Lojasiewicz--Simon inequality for the metric slope. To establish the inequality, we lift the objective to a Hilbert space of random variables and use the analyticity of the lifted gradient in a stronger L^∞ topology to overcome its lack of continuous differentiability in the Hilbert-space topology. Our convergence result does not require global displacement convexity, a Polyak--Lojasiewicz-type condition, or initialization near a minimizer; the objective may remain genuinely nonconvex.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 25 of its 56 references that are in the map. First posted on arXiv 26 Nov 2023.
Builds on
25 of 56 references.
References · 25
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2016Gaussian Error Linear Units (GELUs)cited by 3,502
- 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
- 2017Stable Architectures for Deep Neural Networkscited by 341
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
- 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
- 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
- 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
- 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
13 more
- 2019Machine Learning from a Continuous Viewpointcited by 50
- 2021Two-layer neural networks with values in a Banach spacecited by 26
- 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
- 2019Mean-Field Neural ODEs via Relaxed Optimal Controlcited by 16
- 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 16
- 2020Non-convergence of stochastic gradient descent in the training of deep neural networkscited by 14
- 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 12
- 2021A Measure Theoretical Approach to the Mean-field Maximum Principle for Training NeurODEscited by 11
- 2021A gradient flow equation for optimal control problems with end-point costcited by 6
- 2022Learning via nonlinear conjugate gradients and depth-varying neural ODEscited by 3
- 2023Variational formulations of ODE-Net as a mean-field optimal control problem and existence resultscited by 3
- 2019Intrinsic Lipschitz Regularity of Mean-Field Optimal Controlscited by 1
- 2022Optimal control of nonlocal continuity equations: numerical solutioncited by 1
Built on by
Citing papers per year: 2024: 2; 2025: 0; 2026: 3.
5 papers.
- 2024Extended Flow Matching: a Method of Conditional Generation with Generalized Continuity Equationcited by 11
- 2024Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transportcited by 8
- 2026Man, Machine, and Mathematicscited by 0
- 2026Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNetscited by 0
- 2026Global Exponential Convergence of Two-Layer Linear Network Trainingcited by 0
- 2026Global Exponential Convergence of Two-Layer Linear Network Training7 Oct 2026
- 2026Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets18 Jul 2026
- 2026Man, Machine, and Mathematics29 Apr 2026
- 2024Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transport19 Mar 2024
- 2024Extended Flow Matching: a Method of Conditional Generation with Generalized Continuity Equation29 Feb 2024