mlsanityData to 14 Oct 2026

Discovered Policy Optimisation

arXiv 2210.05639 ↗First posted 11 Oct 2022cs.LG cs.AIcited by 104

Not on the map. Why?

Cited by 50 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Tremendous progress has been made in reinforcement learning (RL) over the past decade. Most of these advancements came through the continual development of new algorithms, which were designed using a combination of mathematical derivations, intuitions, and experimentation. Such an approach of creating algorithms manually is limited by human understanding and ingenuity. In contrast, meta-learning provides a toolkit for automatic machine learning method optimisation, potentially addressing this flaw. However, black-box approaches which attempt to discover RL algorithms with minimal prior structure have thus far not outperformed existing hand-crafted algorithms. Mirror Learning, which includes RL algorithms, such as PPO, offers a potential middle-ground starting point: while every method in this framework comes with theoretical guarantees, components that differentiate them are subject to design. In this paper we explore the Mirror Learning space by meta-learning a "drift" function. We refer to the immediate result as Learnt Policy Optimisation (LPO). By analysing LPO we gain original insights into policy optimisation which we use to formulate a novel, closed-form RL algorithm, Discovered Policy Optimisation (DPO). Our experiments in Brax environments confirm state-of-the-art performance of LPO and DPO, as well as their transfer to unseen settings.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Builds on 28 of its 59 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

28 of 59 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  12. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
16 more
  1. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  2. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  3. 2018Meta-Gradient Reinforcement Learningcited by 159
  4. 2018Evolved Policy Gradientscited by 88
  5. 2021Gradients are Not All You Needcited by 86
  6. 2020Discovering Reinforcement Learning Algorithmscited by 77
  7. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  8. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 69
  9. 2019Truly Proximal Policy Optimizationcited by 69
  10. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  11. 2020Meta Learning Backpropagation And Improving Itcited by 49
  12. 2021Evolving Reinforcement Learning Algorithmscited by 49
  13. 2020Revisiting Design Choices in Proximal Policy Optimizationcited by 38
  14. 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
  15. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  16. 2020Meta-learning curiosity algorithmscited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 15; 2024: 37; 2025: 34; 2026: 16.

104 papers.

Sort:

By area:

  1. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  2. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  3. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
  4. 2024Simplifying Deep Temporal Difference Learningcited by 78
  5. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 63
  6. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  7. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  8. 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 51
  9. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 45
  10. 2022Discovering Evolution Strategies via Meta-Black-Box Optimizationcited by 44
  11. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 42
  12. 2022evosax: JAX-based Evolution Strategiescited by 41
38 more
  1. 2023Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimizationcited by 33
  2. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 31
  3. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 30
  4. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 26
  5. 2025Evolution Strategies at the Hyperscalecited by 25
  6. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  7. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  8. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  9. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 18
  10. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 18
  11. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  12. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  13. 2024A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learningcited by 15
  14. 2023JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for tradingcited by 14
  15. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  16. 2025Improving Transformer World Models for Data-Efficient RLcited by 13
  17. 2023Learning to Optimize for Reinforcement Learningcited by 12
  18. 2024JaxUED: A simple and useable UED library in Jaxcited by 11
  19. 2024Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timestepscited by 11
  20. 2025A Clean Slate for Offline Reinforcement Learningcited by 11
  21. 2023ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantagescited by 10
  22. 2023minimax: Efficient Baselines for Autocurricula in JAXcited by 10
  23. 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learningcited by 10
  24. 2024Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learningcited by 8
  25. 2024EvIL: Evolution Strategies for Generalisable Imitation Learningcited by 8
  26. 2025SocialJax: An Evaluation Suite for Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 8
  27. 2023Deep Model Predictive Optimizationcited by 7
  28. 2024Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancycited by 7
  29. 2024PCGRL+: Scaling, Control and Generalization in Reinforcement Learning Level Generatorscited by 7
  30. 2025Hadamax Encoding: Elevating Performance in Model-Free Ataricited by 7
  31. 2025Relative Entropy Pathwise Policy Optimizationcited by 7
  32. 2025Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domainscited by 7
  33. 2024Recurrent Reinforcement Learning with Memoroidscited by 6
  34. 2024JaxLife: An Open-Ended Agentic Simulatorcited by 6
  35. 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 6
  36. 2024Behaviour Distillationcited by 5
  37. 2025Predicting Multi-Agent Specialization via Task Parallelizabilitycited by 5
  38. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5

50 most cited of 104 All 104 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator
  2. 2024Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guarantees
  3. 2025Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation
  4. 2023Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes
  5. 2018Fast Policy Learning through Imitation and Reinforcement
  6. 2024MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning
  7. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
  8. 2024Reflective Policy Optimization