# Neural Optimizer Search with Reinforcement Learning

arXiv: https://arxiv.org/abs/1709.07417 · HTML: https://mlsanity.com/paper/1709.07417/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2017.

## Built on by (most cited first, 30 of 82)

- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [GLiT: Neural Architecture Search for Global and Local Image Transformer](https://mlsanity.com/paper/2107.02960/) (2021)
- [Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts](https://mlsanity.com/paper/2310.05898/) (2023)
- [Evolving Reinforcement Learning Algorithms](https://mlsanity.com/paper/2101.03958/) (2021)
- [Automated Dynamic Algorithm Configuration](https://mlsanity.com/paper/2205.13881/) (2022)
- [AutoDropout: Learning Dropout Patterns to Regularize Deep Networks](https://mlsanity.com/paper/2101.01761/) (2021)
- [Practical tradeoffs between memory, compute, and performance in learned optimizers](https://mlsanity.com/paper/2203.11860/) (2022)
- [How to Train Your MAML to Excel in Few-Shot Classification](https://mlsanity.com/paper/2106.16245/) (2021)
- [BN-NAS: Neural Architecture Search with Batch Normalization](https://mlsanity.com/paper/2108.07375/) (2021)
- [Finite Expression Method for Solving High-Dimensional Partial Differential Equations](https://mlsanity.com/paper/2206.10121/) (2022)
- [Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization](https://mlsanity.com/paper/2207.13676/) (2022)
- [Symbolic Learning to Optimize: Towards Interpretability and Scalability](https://mlsanity.com/paper/2203.06578/) (2022)
- [MSINet: Twins Contrastive Search of Multi-Scale Interaction for Object ReID](https://mlsanity.com/paper/2303.07065/) (2023)
- [G-CoS: GNN-Accelerator Co-Search Towards Both Better Accuracy and Efficiency](https://mlsanity.com/paper/2109.08983/) (2021)
- [B2Opt: Learning to Optimize Black-box Optimization with Little Budget](https://mlsanity.com/paper/2304.11787/) (2023)
- [Improving exploration in policy gradient search: Application to symbolic optimization](https://mlsanity.com/paper/2107.09158/) (2021)
- [Training Learned Optimizers with Randomly Initialized Learned Optimizers](https://mlsanity.com/paper/2101.07367/) (2021)
- [Acceleration via Fractal Learning Rate Schedules](https://mlsanity.com/paper/2103.01338/) (2021)
- [Selecting and Composing Learning Rate Policies for Deep Neural Networks](https://mlsanity.com/paper/2210.12936/) (2022)
- [T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging](https://mlsanity.com/paper/2111.07535/) (2021)
- [Learning to Optimize in Model Predictive Control](https://mlsanity.com/paper/2212.02603/) (2022)
- [AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement](https://mlsanity.com/paper/2608.20318/) (2026)
- [AutoLossGen: Automatic Loss Function Generation for Recommender Systems](https://mlsanity.com/paper/2204.13160/) (2022)
- [Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaboration](https://mlsanity.com/paper/2410.08102/) (2024)
- [MetricOpt: Learning to Optimize Black-Box Evaluation Metrics](https://mlsanity.com/paper/2104.10631/) (2021)
- [Graph-based Neural Acceleration for Nonnegative Matrix Factorization](https://mlsanity.com/paper/2202.00264/) (2022)
- [Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars](https://mlsanity.com/paper/2211.01842/) (2022)

## Abstract (arXiv)

We present an approach to automate the process of discovering optimization methods, with a focus on deep learning architectures. We train a Recurrent Neural Network controller to generate a string in a domain specific language that describes a mathematical update equation based on a list of primitive functions, such as the gradient, running average of the gradient, etc. The controller is trained with Reinforcement Learning to maximize the performance of a model after a few epochs. On CIFAR-10, our method discovers several update rules that are better than many commonly used optimizers, such as Adam, RMSProp, or SGD with and without Momentum on a ConvNet model. We introduce two new optimizers, named PowerSign and AddSign, which we show transfer well and improve training on a variety of different tasks and architectures, including ImageNet classification and Google's neural machine translation system.