mlsanityData to 7 Oct 2026

On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport

arXiv 1805.09545 ↗First posted 24 May 2018math.OC cs.NE stat.MLcited by 422

Not on the map. Why?

Abstract (arXiv)

Many tasks in machine learning and signal processing can be solved by minimizing a convex function of a measure. This includes sparse spikes deconvolution or training a neural network with a single hidden layer. For these problems, we study a simple minimization method: the unknown measure is discretized into a mixture of particles and a continuous-time gradient descent is performed on their weights and positions. This is an idealization of the usual way to train neural networks with a large hidden layer. We show that, when initialized correctly and in the many-particle limit, this gradient flow, although non-convex, converges to global minimizers. The proof involves Wasserstein gradient flows, a by-product of optimal transport theory. Numerical experiments show that this asymptotic behavior is already at play for a reasonable number of particles, even in high dimension.

Key facts. Cited by 422 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 74; 2022: 74; 2023: 81; 2024: 75; 2025: 64; 2026: 54.

422 papers.

Sort:

By area:

  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  2. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  3. 2023A mathematical perspective on Transformerscited by 160
  4. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  5. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  6. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  7. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  8. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  9. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  10. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  11. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  12. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
38 more
  1. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  2. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  3. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  4. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  5. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  6. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  7. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  8. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  9. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  10. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  11. 2023Learning time-scales in two-layers neural networkscited by 57
  12. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 53
  13. 2021Do Input Gradients Highlight Discriminative Features?cited by 52
  14. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  15. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  16. 2022Learning threshold neurons via the "edge of stability"cited by 50
  17. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  18. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  19. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  20. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  21. 2022Unbalanced Optimal Transport, from Theory to Numericscited by 45
  22. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  23. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  24. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  25. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  26. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  27. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  28. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  29. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  30. 2021LEADS: Learning Dynamical Systems that Generalize Across Environmentscited by 39
  31. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  32. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  33. 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
  34. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  35. 2021KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Supportcited by 37
  36. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  37. 2023Deep networks for system identification: a Surveycited by 35
  38. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35

50 most cited of 422 All 422 (JSON)