# Reinforcement learning to choose optimizers

arXiv: https://arxiv.org/abs/2609.01811 · HTML: https://mlsanity.com/paper/2609.01811/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 40 of its 71 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 40)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Reinforcement learning](https://mlsanity.com/paper/2405.10369/) (2024)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [Efficient and Modular Implicit Differentiation](https://mlsanity.com/paper/2105.15183/) (2021)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [Improving Generalization Performance by Switching from Adam to SGD](https://mlsanity.com/paper/1712.07628/) (2017)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Automated Algorithm Selection: Survey and Perspectives](https://mlsanity.com/paper/1811.11597/) (2018)
- [Understanding and correcting pathologies in the training of learned optimizers](https://mlsanity.com/paper/1810.10180/) (2018)
- [Decoupling Value and Policy for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2102.10330/) (2021)
- [Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians](https://mlsanity.com/paper/1901.08244/) (2019)
- [evosax: JAX-based Evolution Strategies](https://mlsanity.com/paper/2212.04180/) (2022)
- [Automated Dynamic Algorithm Configuration](https://mlsanity.com/paper/2205.13881/) (2022)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [Exploratory Landscape Analysis is Strongly Sensitive to the Sampling Strategy](https://mlsanity.com/paper/2006.11135/) (2020)
- [Deep Reinforcement Learning for Dynamic Algorithm Selection: A Proof-of-Principle Study on Differential Evolution](https://mlsanity.com/paper/2403.02131/) (2024)
- [Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimization](https://mlsanity.com/paper/2411.00625/) (2024)
- [Per-run Algorithm Selection with Warm-starting using Trajectory-based Features](https://mlsanity.com/paper/2204.09483/) (2022)
- [ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning](https://mlsanity.com/paper/2412.07507/) (2024)
- [Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking](https://mlsanity.com/paper/2010.04542/) (2020)
- [Meta Learning Black-Box Population-Based Optimizers](https://mlsanity.com/paper/2103.03526/) (2021)
- [DoE2Vec: Deep-learning Based Features for Exploratory Landscape Analysis](https://mlsanity.com/paper/2304.01219/) (2023)
- [The Impact of Hyper-Parameter Tuning for Landscape-Aware Performance Regression and Algorithm Selection](https://mlsanity.com/paper/2104.09272/) (2021)
- [To Switch or not to Switch: Predicting the Benefit of Switching between Algorithms based on Trajectory Features](https://mlsanity.com/paper/2302.09075/) (2023)

## Abstract (arXiv)

No single optimization method is uniformly best for all problems, and the most suitable optimizer choice can change during a run. Existing approaches that change optimizer during execution typically predetermine part of the strategy: the portfolio is restricted to one algorithm class, the switch occurs once at a fixed time, or the frequency of decisions is treated as a hyperparameter rather than a learned one. We introduce "Reinforcement Learning to Choose Optimizers", which formulates the optimization algorithm choice as a sequential decision-making problem. At each decision, a recurrent policy reads the current run state and decides both which optimizer should be used next and for how long. The portfolio includes both gradient-based and derivative-free optimizers, and each switch passes on the current best solution and a representative step size. A context proxy conditions a gating network over expert heads, and training employs a decoupled actor-critic whose return is expressed in the same empirical runtime distribution metric used at evaluation. Training tasks and portfolio are designed jointly so that no optimizer dominates. On unseen problems, the learned policy outperforms every portfolio optimizer at all but the smallest budgets, and it remains robust under distribution shift.