# Global L2 minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learning

arXiv: https://arxiv.org/abs/2311.15487 · HTML: https://mlsanity.com/paper/2311.15487/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 7 of 7)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [How to Start Training: The Effect of Initialization and Architecture](https://mlsanity.com/paper/1803.01719/) (2018)
- [The loss landscape of overparameterized neural networks](https://mlsanity.com/paper/1804.10200/) (2018)
- [On the Benefit of Width for Neural Networks: Disappearance of Bad Basins](https://mlsanity.com/paper/1812.11039/) (2018)
- [When Expressivity Meets Trainability: Fewer than n Neurons Can Work](https://mlsanity.com/paper/2210.12001/) (2022)
- [Which Minimizer Does My Neural Network Converge To?](https://mlsanity.com/paper/2011.02408/) (2020)
- [Geometric structure of shallow neural networks and constructive mathcal L2 cost minimization](https://mlsanity.com/paper/2309.10370/) (2023)

## Built on by (most cited first, 2 of 2)

- [Gradient flow in parameter space is equivalent to linear interpolation in output space](https://mlsanity.com/paper/2408.01517/) (2024)
- [Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning](https://mlsanity.com/paper/2501.07400/) (2025)

## Abstract (arXiv)

We consider the scenario of supervised learning in Deep Learning (DL) networks, and exploit the arbitrariness of choice in the Riemannian metric relative to which the gradient descent flow can be defined (a general fact of differential geometry). In the standard approach to DL, the gradient flow on the space of parameters (weights and biases) is defined with respect to the Euclidean metric. Here instead, we choose the gradient flow with respect to the Euclidean metric in the output layer of the DL network. This naturally induces two modified versions of the gradient descent flow in the parameter space, one adapted for the overparametrized setting, and the other for the underparametrized setting. In the overparametrized case, we prove that, provided that a rank condition holds, all orbits of the modified gradient descent drive the mathcal L2 cost to its global minimum at a uniform exponential convergence rate; one thereby obtains an a priori stopping time for any prescribed proximity to the global minimum. We point out relations of the latter to sub-Riemannian geometry. Moreover, we generalize the above framework to the situation in which the rank condition does not hold; in particular, we show that local equilibria can only exist if a rank loss occurs, and that generically, they are not isolated points, but elements of a critical submanifold of parameter space.