# Learning Gradient Descent: Better Generalization and Longer Horizons

arXiv: https://arxiv.org/abs/1703.03633 · HTML: https://mlsanity.com/paper/1703.03633/

## Key facts

- Cited by 50 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 10 Mar 2017.

## Built on by (most cited first, 30 of 50)

- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Learning to Learn with Generative Models of Neural Network Checkpoints](https://mlsanity.com/paper/2209.12892/) (2022)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [Gradients are Not All You Need](https://mlsanity.com/paper/2111.05803/) (2021)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [Bootstrapped Meta-Learning](https://mlsanity.com/paper/2109.04504/) (2021)
- [Automated Dynamic Algorithm Configuration](https://mlsanity.com/paper/2205.13881/) (2022)
- [Practical tradeoffs between memory, compute, and performance in learned optimizers](https://mlsanity.com/paper/2203.11860/) (2022)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [Gradient-based Bi-level Optimization for Deep Learning: A Survey](https://mlsanity.com/paper/2207.11719/) (2022)
- [Improving physics-informed neural networks with meta-learned optimization](https://mlsanity.com/paper/2303.07127/) (2023)
- [Symbolic Learning to Optimize: Towards Interpretability and Scalability](https://mlsanity.com/paper/2203.06578/) (2022)
- [Towards Constituting Mathematical Structures for Learning to Optimize](https://mlsanity.com/paper/2305.18577/) (2023)
- [Learn2Hop: Learned Optimization on Rough Landscapes](https://mlsanity.com/paper/2107.09661/) (2021)
- [Learning to Optimize for Reinforcement Learning](https://mlsanity.com/paper/2302.01470/) (2023)
- [Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Process](https://mlsanity.com/paper/2401.03244/) (2024)
- [Learning to Optimize in Model Predictive Control](https://mlsanity.com/paper/2212.02603/) (2022)
- [A Simple Guard for Learned Optimizers](https://mlsanity.com/paper/2201.12426/) (2022)
- [Mnemosyne: Learning to Train Transformers with Transformers](https://mlsanity.com/paper/2302.01128/) (2023)
- [Learning to Generalize Provably in Learning to Optimize](https://mlsanity.com/paper/2302.11085/) (2023)
- [Deep Model Predictive Optimization](https://mlsanity.com/paper/2310.04590/) (2023)
- [Stochastic Unrolled Federated Learning](https://mlsanity.com/paper/2305.15371/) (2023)
- [Learning to Optimize with Dynamic Mode Decomposition](https://mlsanity.com/paper/2211.16268/) (2022)
- [M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation](https://mlsanity.com/paper/2303.00039/) (2023)
- [Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness](https://mlsanity.com/paper/2110.08256/) (2021)
- [Learning to Optimize Quasi-Newton Methods](https://mlsanity.com/paper/2210.06171/) (2022)
- [Semantic are Beacons: A Semantic Perspective for Unveiling Parameter-Efficient Fine-Tuning in Knowledge Learning](https://mlsanity.com/paper/2405.18292/) (2024)
- [Narrowing the Focus: Learned Optimizers for Pretrained Models](https://mlsanity.com/paper/2408.09310/) (2024)
- [Efficient End-to-End Learning for Decision-Making: A Meta-Optimization Approach](https://mlsanity.com/paper/2505.11360/) (2025)
- [Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs](https://mlsanity.com/paper/2510.00419/) (2025)

## Abstract (arXiv)

Training deep neural networks is a highly nontrivial task, involving carefully selecting appropriate training algorithms, scheduling step sizes and tuning other hyperparameters. Trying different combinations can be quite labor-intensive and time consuming. Recently, researchers have tried to use deep learning algorithms to exploit the landscape of the loss function of the training problem of interest, and learn how to optimize over it in an automatic way. In this paper, we propose a new learning-to-learn model and some useful and practical tricks. Our optimizer outperforms generic, hand-crafted optimization algorithms and state-of-the-art learning-to-learn optimizers by DeepMind in many tasks. We demonstrate the effectiveness of our algorithms on a number of tasks, including deep MLPs, CNNs, and simple LSTMs.