# Learning to Optimize Neural Nets

arXiv: https://arxiv.org/abs/1703.00441 · HTML: https://mlsanity.com/paper/1703.00441/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2017.

## Built on by (most cited first, 30 of 41)

- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Learning to Learn with Generative Models of Neural Network Checkpoints](https://mlsanity.com/paper/2209.12892/) (2022)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [MetaAlign: Coordinating Domain Alignment and Classification for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2103.13575/) (2021)
- [Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies](https://mlsanity.com/paper/2112.13835/) (2021)
- [Meta-Inverse Reinforcement Learning with Probabilistic Context Variables](https://mlsanity.com/paper/1909.09314/) (2019)
- [Learning a Prior over Intent via Meta-Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.12573/) (2018)
- [Practical tradeoffs between memory, compute, and performance in learned optimizers](https://mlsanity.com/paper/2203.11860/) (2022)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [Symbolic Learning to Optimize: Towards Interpretability and Scalability](https://mlsanity.com/paper/2203.06578/) (2022)
- [Learning to Optimize in Model Predictive Control](https://mlsanity.com/paper/2212.02603/) (2022)
- [Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Process](https://mlsanity.com/paper/2401.03244/) (2024)
- [QuickSplat: Fast 3D Surface Reconstruction via Learned Gaussian Initialization](https://mlsanity.com/paper/2505.05591/) (2025)
- [Amortized Proximal Optimization](https://mlsanity.com/paper/2203.00089/) (2022)
- [Large Language Models for Human-Machine Collaborative Particle Accelerator Tuning through Natural Language](https://mlsanity.com/paper/2405.08888/) (2024)
- [Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Single](https://mlsanity.com/paper/2304.11153/) (2023)
- [Deep Model Predictive Optimization](https://mlsanity.com/paper/2310.04590/) (2023)
- [Learning to optimize with convergence guarantees using nonlinear system theory](https://mlsanity.com/paper/2403.09389/) (2024)
- [A Nonstochastic Control Approach to Optimization](https://mlsanity.com/paper/2301.07902/) (2023)
- [Learning To Optimize Quantum Neural Network Without Gradients](https://mlsanity.com/paper/2304.07442/) (2023)
- [Learning to Optimize Quasi-Newton Methods](https://mlsanity.com/paper/2210.06171/) (2022)
- [Learning to Generate Gradients for Test-Time Adaptation via Test-Time Training Layers](https://mlsanity.com/paper/2412.16901/) (2024)
- [Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs](https://mlsanity.com/paper/2510.00419/) (2025)
- [Federated Automatic Differentiation](https://mlsanity.com/paper/2301.07806/) (2023)
- [Learning to Do or Learning While Doing: Reinforcement Learning and Bayesian Optimisation for Online Continuous Tuning](https://mlsanity.com/paper/2306.03739/) (2023)
- [Improving Learning to Optimize Using Parameter Symmetries](https://mlsanity.com/paper/2504.15399/) (2025)
- [Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective](https://mlsanity.com/paper/2505.19815/) (2025)
- [A Semi-amortized Lifted Learning-to-Optimize Masked (SALLO-M) Transformer Model for Scalable and Generalizable Beamforming](https://mlsanity.com/paper/2510.13077/) (2025)
- [Efficient Meta Subspace Optimization](https://mlsanity.com/paper/2110.14920/) (2021)

## Abstract (arXiv)

Learning to Optimize is a recently proposed framework for learning optimization algorithms using reinforcement learning. In this paper, we explore learning an optimization algorithm for training shallow neural nets. Such high-dimensional stochastic optimization problems present interesting challenges for existing reinforcement learning algorithms. We develop an extension that is suited to learning optimization algorithms in this setting and demonstrate that the learned optimization algorithm consistently outperforms other known optimization algorithms even on unseen tasks and is robust to changes in stochasticity of gradients and the neural net architecture. More specifically, we show that an optimization algorithm trained with the proposed method on the problem of training a neural net on MNIST generalizes to the problems of training neural nets on the Toronto Faces Dataset, CIFAR-10 and CIFAR-100.