# A qualitative difference between gradient flows of convex functions in finite- and infinite-dimensional Hilbert spaces

arXiv: https://arxiv.org/abs/2310.17610 · HTML: https://mlsanity.com/paper/2310.17610/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Convex Optimization: Algorithms and Complexity](https://mlsanity.com/paper/1405.4980/) (2014)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond](https://mlsanity.com/paper/1611.07476/) (2016)
- [Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks](https://mlsanity.com/paper/2006.07322/) (2020)
- [Spurious Local Minima are Common in Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/1712.08968/) (2017)
- [A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics](https://mlsanity.com/paper/1904.04326/) (2019)
- [Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis](https://mlsanity.com/paper/2105.01650/) (2021)
- [On exponential convergence of SGD in non-convex over-parametrized learning](https://mlsanity.com/paper/1811.02564/) (2018)
- [The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutions](https://mlsanity.com/paper/2006.05900/) (2020)
- [Spurious Valleys in Two-layer Neural Network Optimization Landscapes](https://mlsanity.com/paper/1802.06384/) (2018)
- [Negative eigenvalues of the Hessian in deep neural networks](https://mlsanity.com/paper/1902.02366/) (2019)
- [The rate of convergence of Nesterov's accelerated forward-backward method is actually faster than 1/k2](https://mlsanity.com/paper/1510.08740/) (2015)
- [Optimal convergence rates for Nesterov acceleration](https://mlsanity.com/paper/1805.05719/) (2018)
- [Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networks](https://mlsanity.com/paper/2107.01832/) (2021)
- [Nesterov acceleration despite very noisy gradients](https://mlsanity.com/paper/2302.05515/) (2023)
- [Path Length Bounds for Gradient Descent and Flow](https://mlsanity.com/paper/1908.01089/) (2019)
- [Accelerated First-Order Methods: Differential Equations and Lyapunov Functions](https://mlsanity.com/paper/1903.05671/) (2019)
- [A convex function satisfying the Lojasiewicz inequality but failing the gradient conjecture both at zero and infinity](https://mlsanity.com/paper/2102.05342/) (2021)

## Built on by (most cited first, 3 of 3)

- [Towards understanding Accelerated Stein Variational Gradient Flow -- Analysis of Generalized Bilinear Kernels for Gaussian target distributions](https://mlsanity.com/paper/2509.04008/) (2025)
- [Momentum-based minimization of the Ginzburg-Landau functional on Euclidean spaces and graphs](https://mlsanity.com/paper/2501.00389/) (2024)
- [Sharp Convergence of Wasserstein Gradient Flows for Spectrally Nonnegative Interaction Energies](https://mlsanity.com/paper/2609.27008/) (2026)

## Abstract (arXiv)

We consider gradient flow/gradient descent and heavy ball/accelerated gradient descent optimization for convex objective functions. In the gradient flow case, we prove the following: 1. If f does not have a minimizer, the convergence f(xt)→ inf f can be arbitrarily slow. 2. If f does have a minimizer, the excess energy f(xt) - inf f is integrable/summable in time. In particular, f(xt) - inf f = o(1/t) as t→∞. 3. In Hilbert spaces, this is optimal: f(xt) - inf f can decay to 0 as slowly as any given function which is monotone decreasing and integrable at ∞, even for a fixed quadratic objective. 4. In finite dimension (or more generally, for all gradient flow curves of finite length), this is not optimal: We prove that there are convex monotone decreasing integrable functions g(t) which decrease to zero slower than f(xt)-inf f for the gradient flow of any convex function on mathbb Rd. For instance, we show that any gradient flow xt of a convex function f in finite dimension satisfies liminft→∞ big(t· log2(t)· bigf(xt) -inf fbigbig)=0. This improves on the commonly reported O(1/t) rate and provides a sharp characterization of the energy decay law. We also note that it is impossible to establish a rate O(1/(tφ(t)) for any function φ which satisfies limt→∞φ(t) = ∞, even asymptotically. Similar results are obtained in related settings for (1) discrete time gradient descent, (2) stochastic gradient descent with multiplicative noise and (3) the heavy ball ODE. In the case of stochastic gradient descent, the summability of mathbb E[f(xn) - inf f] is used to prove that f(xn)→ inf f almost surely - an improvement on the convergence almost surely up to a subsequence which follows from the O(1/n) decay estimate.