# A convergence result of a continuous model of deep learning via a Lojasiewicz--Simon inequality

arXiv: https://arxiv.org/abs/2311.15365 · HTML: https://mlsanity.com/paper/2311.15365/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 26 Nov 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019)
- [Loss landscapes and optimization in over-parameterized non-linear systems and neural networks](https://mlsanity.com/paper/2003.00307/) (2020)
- [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019)
- [A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth](https://mlsanity.com/paper/2003.05508/) (2020)
- [Sparse Optimization on Measures with Over-parameterized Gradient Descent](https://mlsanity.com/paper/1907.10300/) (2019)
- [Machine Learning from a Continuous Viewpoint](https://mlsanity.com/paper/1912.12777/) (2019)
- [Two-layer neural networks with values in a Banach space](https://mlsanity.com/paper/2105.02095/) (2021)
- [Overparameterization of deep ResNet: zero loss and mean-field analysis](https://mlsanity.com/paper/2105.14417/) (2021)
- [Mean-Field Neural ODEs via Relaxed Optimal Control](https://mlsanity.com/paper/1912.05475/) (2019)
- [Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalization](https://mlsanity.com/paper/2110.08084/) (2021)
- [Non-convergence of stochastic gradient descent in the training of deep neural networks](https://mlsanity.com/paper/2006.07075/) (2020)
- [On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime](https://mlsanity.com/paper/2110.02926/) (2021)
- [A Measure Theoretical Approach to the Mean-field Maximum Principle for Training NeurODEs](https://mlsanity.com/paper/2107.08707/) (2021)
- [A gradient flow equation for optimal control problems with end-point cost](https://mlsanity.com/paper/2107.00556/) (2021)
- [Learning via nonlinear conjugate gradients and depth-varying neural ODEs](https://mlsanity.com/paper/2202.05766/) (2022)
- [Variational formulations of ODE-Net as a mean-field optimal control problem and existence results](https://mlsanity.com/paper/2303.05924/) (2023)
- [Intrinsic Lipschitz Regularity of Mean-Field Optimal Controls](https://mlsanity.com/paper/1908.04183/) (2019)
- [Optimal control of nonlocal continuity equations: numerical solution](https://mlsanity.com/paper/2212.06608/) (2022)

## Built on by (most cited first, 5 of 5)

- [Extended Flow Matching: a Method of Conditional Generation with Generalized Continuity Equation](https://mlsanity.com/paper/2402.18839/) (2024)
- [Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transport](https://mlsanity.com/paper/2403.12887/) (2024)
- [Man, Machine, and Mathematics](https://mlsanity.com/paper/2604.27052/) (2026)
- [Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets](https://mlsanity.com/paper/2607.16761/) (2026)
- [Global Exponential Convergence of Two-Layer Linear Network Training](https://mlsanity.com/paper/2610.09356/) (2026)

## Abstract (arXiv)

We study an idealized training process for deep neural networks in a continuous-depth, mean-field model in which each layer is parameterized by a probability measure on a Euclidean parameter space. The training dynamics are formulated as a Wasserstein-type gradient flow of an objective with a fixed L2-regularization. Under suitable analyticity and growth assumptions, together with a coercivity assumption and sufficient regularity of the initial data, we prove that every curve of maximal slope converges to a single critical point of the objective as the training time tends to infinity. The proof combines compactness of the curve with a Lojasiewicz--Simon inequality for the metric slope. To establish the inequality, we lift the objective to a Hilbert space of random variables and use the analyticity of the lifted gradient in a stronger L^∞ topology to overcome its lack of continuous differentiability in the Hilbert-space topology. Our convergence result does not require global displacement convexity, a Polyak--Lojasiewicz-type condition, or initialization near a minimizer; the objective may remain genuinely nonconvex.