# Learning Provably Improves the Convergence of Gradient Descent

arXiv: https://arxiv.org/abs/2501.18092 · HTML: https://mlsanity.com/paper/2501.18092/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 30 Jan 2025.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Loss landscapes and optimization in over-parameterized non-linear systems and neural networks](https://mlsanity.com/paper/2003.00307/) (2020)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Learning to Optimize: Training Deep Neural Networks for Wireless Resource Management](https://mlsanity.com/paper/1705.09412/) (2017)
- [Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds](https://mlsanity.com/paper/1808.10038/) (2018)
- [On the Convergence Rate of Training Recurrent Neural Networks](https://mlsanity.com/paper/1810.12065/) (2018)
- [Graph Neural Networks for Scalable Radio Resource Management: Architecture Design and Theoretical Analysis](https://mlsanity.com/paper/2007.07632/) (2020)
- [Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology](https://mlsanity.com/paper/2002.07867/) (2020)
- [Learning Gradient Descent: Better Generalization and Longer Horizons](https://mlsanity.com/paper/1703.03633/) (2017)
- [Iterative Algorithm Induced Deep-Unfolding Neural Networks: Precoding Design for Multiuser MIMO Systems](https://mlsanity.com/paper/2006.08099/) (2020)
- [On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths](https://mlsanity.com/paper/2101.09612/) (2021)
- [Safeguarded Learned Convex Optimization](https://mlsanity.com/paper/2003.01880/) (2020)
- [Hyperparameter Tuning is All You Need for LISTA](https://mlsanity.com/paper/2110.15900/) (2021)
- [Towards Constituting Mathematical Structures for Learning to Optimize](https://mlsanity.com/paper/2305.18577/) (2023)
- [Exploring the loss landscape of regularized neural networks via convex duality](https://mlsanity.com/paper/2411.07729/) (2024)
- [Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time](https://mlsanity.com/paper/2402.03625/) (2024)
- [Learning to Generalize Provably in Learning to Optimize](https://mlsanity.com/paper/2302.11085/) (2023)
- [Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networks](https://mlsanity.com/paper/2107.01832/) (2021)

## Abstract (arXiv)

Learn to Optimize (L2O) trains deep neural network-based solvers for optimization, achieving success in accelerating convex problems and improving non-convex solutions. However, L2O lacks rigorous theoretical backing for its own training convergence, as existing analyses often use unrealistic assumptions -- a gap this work highlights empirically. We bridge this gap by proving the training convergence of L2O models that learn Gradient Descent (GD) hyperparameters for quadratic programming, leveraging the Neural Tangent Kernel (NTK) theory. We propose a deterministic initialization strategy to support our theoretical results and promote stable training over extended optimization horizons by mitigating gradient explosion. Our L2O framework demonstrates over 50% better optimality than GD and superior robustness over state-of-the-art L2O methods on synthetic datasets. The code of our method can be found from https://github.com/NetX-lab/MathL2OProof-Official.