# Learning to Learn without Gradient Descent by Gradient Descent

arXiv: https://arxiv.org/abs/1611.03824 · HTML: https://mlsanity.com/paper/1611.03824/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2016.

## Built on by (most cited first, 9 of 9)

- [Bootstrapped Meta-Learning](https://mlsanity.com/paper/2109.04504/) (2021)
- [Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art](https://mlsanity.com/paper/2308.16316/) (2023)
- [Mnemosyne: Learning to Train Transformers with Transformers](https://mlsanity.com/paper/2302.01128/) (2023)
- [L2NAS: Learning to Optimize Neural Architectures via Continuous-Action Reinforcement Learning](https://mlsanity.com/paper/2109.12425/) (2021)
- [An Investigation of the Bias-Variance Tradeoff in Meta-Gradients](https://mlsanity.com/paper/2209.11303/) (2022)
- [ODE-based Learning to Optimize](https://mlsanity.com/paper/2406.02006/) (2024)
- [Efficient End-to-End Learning for Decision-Making: A Meta-Optimization Approach](https://mlsanity.com/paper/2505.11360/) (2025)
- [Optimistic Meta-Gradients](https://mlsanity.com/paper/2301.03236/) (2023)
- [Learning Instance-Specific Parameters of Black-Box Models Using Differentiable Surrogates](https://mlsanity.com/paper/2407.17530/) (2024)

## Abstract (arXiv)

We learn recurrent neural network optimizers trained on simple synthetic functions by gradient descent. We show that these learned optimizers exhibit a remarkable degree of transfer in that they can be used to efficiently optimize a broad range of derivative-free black-box functions, including Gaussian process bandits, simple control objectives, global optimization benchmarks and hyper-parameter tuning tasks. Up to the training horizon, the learned optimizers learn to trade-off exploration and exploitation, and compare favourably with heavily engineered Bayesian optimization packages for hyper-parameter tuning.