mlsanityData to 7 Oct 2026

Can Learned Optimization Make Reinforcement Learning Less Difficult?

arXiv 2407.07082 ↗First posted 9 Jul 2024cs.LG cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

While reinforcement learning (RL) holds great potential for decision making in the real world, it suffers from a number of unique difficulties which often need specific consideration. In particular: it is highly non-stationary; suffers from high degrees of plasticity loss; and requires exploration to prevent premature convergence to local optima and maximize return. In this paper, we consider whether learned optimization can help overcome these problems. Our method, Learned Optimization for Plasticity, Exploration and Non-stationarity (OPEN), meta-learns an update rule whose input features and output structure are informed by previously proposed solutions to these difficulties. We show that our parameterization is flexible enough to enable meta-learning in diverse learning contexts, including the ability to use stochasticity for exploration. Our experiments demonstrate that when meta-trained on single and small sets of environments, OPEN outperforms or equals traditionally used optimizers. Furthermore, OPEN shows strong generalization characteristics across a range of environments and agent architectures.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 63 of its 90 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

63 of 90 references.

References · 63

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Layer Normalizationcited by 5,579
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  9. 2019On the Convergence of Adam and Beyondcited by 834
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  12. 2016Learning to learn by gradient descent by gradient descentcited by 616
38 more
  1. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  2. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  3. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  4. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  5. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  6. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  7. 2017Count-Based Exploration with Neural Density Modelscited by 255
  8. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  9. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  10. 2016Learning to Optimizecited by 195
  11. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  12. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  13. 2023Understanding plasticity in neural networkscited by 170
  14. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  15. 2017Parameter Space Noise for Explorationcited by 158
  16. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  17. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
  18. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  19. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  20. 2019Behaviour Suite for Reinforcement Learningcited by 129
  21. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  22. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  23. 2017Learned Optimizers that Scale and Generalizecited by 109
  24. 2022Discovered Policy Optimisationcited by 103
  25. 2023A Tutorial on Meta-Reinforcement Learningcited by 102
  26. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  27. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  28. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  29. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  30. 2020Discovering Reinforcement Learning Algorithmscited by 75
  31. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  32. 2024Simplifying Deep Temporal Difference Learningcited by 71
  33. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  34. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 57
  35. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  36. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  37. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  38. 2022evosax: JAX-based Evolution Strategiescited by 39

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  2. 2025Evolution Strategies at the Hyperscalecited by 22
  3. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  4. 2025Vintix: Action Model via In-Context Reinforcement Learningcited by 8
  5. 2026Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMscited by 6
  6. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
  7. 2025Celo: Training Versatile Learned Optimizers on a Compute Dietcited by 4
  8. 2024Meta-Learning Objectives for Preference Optimizationcited by 2
  9. 2025On Discovering Algorithms for Adversarial Imitation Learningcited by 1
  10. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  11. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  12. 2024A Research Agenda for Usability and Generalisation in Reinforcement Learningcited by 0
2 more
  1. 2026POP: Prior-Fitted First-Order Optimization Policiescited by 0
  2. 2026The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamicscited by 0