mlsanityData to 7 Oct 2026

Learning to Optimize for Reinforcement Learning

arXiv 2302.01470 ↗First posted 3 Feb 2023cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

In recent years, by leveraging more data, computation, and diverse tasks, learned optimizers have achieved remarkable success in supervised learning, outperforming classical hand-designed optimizers. Reinforcement learning (RL) is essentially different from supervised learning, and in practice, these learned optimizers do not work well even in simple RL tasks. We investigate this phenomenon and identify two issues. First, the agent-gradient distribution is non-independent and identically distributed, leading to inefficient meta-training. Moreover, due to highly stochastic agent-environment interactions, the agent-gradients have high bias and variance, which increases the difficulty of learning an optimizer for RL. We propose pipeline training and a novel optimizer structure with a good inductive bias to address these issues, making it possible to learn an optimizer for reinforcement learning from scratch. We show that, although only trained in toy tasks, our learned optimizer can generalize to unseen complex tasks in Brax.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

42 of 59 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  8. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  9. 2016Learning to learn by gradient descent by gradient descentcited by 616
  10. 2016Error bounds for approximations with deep ReLU networkscited by 567
  11. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  12. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
30 more
  1. 2016Learning to Optimizecited by 195
  2. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  3. 2019Behaviour Suite for Reinforcement Learningcited by 129
  4. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  5. 2017Learned Optimizers that Scale and Generalizecited by 109
  6. 2022Discovered Policy Optimisationcited by 103
  7. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  8. 2018Evolved Policy Gradientscited by 86
  9. 2020Rational neural networkscited by 75
  10. 2020Discovering Reinforcement Learning Algorithmscited by 75
  11. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  12. 2019Meta-Learning via Learned Losscited by 60
  13. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 57
  14. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  15. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  16. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  17. 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
  18. 2020Interference and Generalization in Temporal Difference Learningcited by 41
  19. 2017Neural networks and rational functionscited by 40
  20. 2018Aggregated Momentum: Stability Through Passive Dampingcited by 35
  21. 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
  22. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  23. 2020Using a thousand optimization tasks to learn hyperparameter search strategiescited by 29
  24. 2021Introducing Symmetries to Black Box Meta Reinforcement Learningcited by 25
  25. 2020Reverse engineering learned optimizers reveals known and novel mechanismscited by 19
  26. 2018Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methodscited by 17
  27. 2017Reinforcement Learning for Learning Rate Controlcited by 16
  28. 2021Correcting Momentum in Temporal Difference Learningcited by 12
  29. 2021Model-free Policy Learning with Reward Gradientscited by 9
  30. 2019Correlation Priors for Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 4; 2026: 3.

12 papers.

  1. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  2. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  3. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  4. 2024Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timestepscited by 11
  5. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
  6. 2025Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspectivecited by 2
  7. 2026AI Training Manager: Bounded Closed-Loop Control of Adaptive Training Recipescited by 1
  8. 2024Survival Multiarmed Bandits with Bootstrapping Methodscited by 0
  9. 2025GNN-DT: Graph Neural Network Enhanced Decision Transformer for Efficient Optimization in Dynamic Environmentscited by 0
  10. 2025ANO : Faster is Better in Noisy Landscapecited by 0
  11. 2026POP: Prior-Fitted First-Order Optimization Policiescited by 0
  12. 2026ReBRAC-v2: The Return of the Kingcited by 0