mlsanityData to 7 Oct 2026

Learning mirror maps in policy mirror descent

arXiv 2402.05187 ↗First posted 7 Feb 2024stat.ML cs.LG math.OCcited by 1

Not on the map. Why?

Abstract (arXiv)

Policy Mirror Descent (PMD) is a popular framework in reinforcement learning, serving as a unifying perspective that encompasses numerous algorithms. These algorithms are derived through the selection of a mirror map and enjoy finite-time convergence guarantees. Despite its popularity, the exploration of PMD's full potential is limited, with the majority of research focusing on a particular mirror map -- namely, the negative entropy -- which gives rise to the renowned Natural Policy Gradient (NPG) method. It remains uncertain from existing theoretical studies whether the choice of mirror map significantly influences PMD's efficacy. In our work, we conduct empirical investigations to show that the conventional mirror map choice (NPG) often yields less-than-optimal outcomes across several standard benchmark environments. Using evolutionary strategies, we identify more efficient mirror maps that enhance the performance of PMD. We first focus on a tabular environment, i.e. Grid-World, where we relate existing theoretical bounds with the performance of PMD for a few standard mirror maps and the learned one. We then show that it is possible to learn a mirror map that outperforms the negative entropy in more complex environments, such as the MinAtar suite. Additionally, we demonstrate that the learned mirror maps generalize effectively to different tasks by testing each map across various other environments.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

39 of 58 references.

References · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  8. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  9. 2014Convex Optimization: Algorithms and Complexitycited by 569
  10. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  11. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 407
  12. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
27 more
  1. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  2. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  3. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  4. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  5. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
  6. 2019Behaviour Suite for Reinforcement Learningcited by 129
  7. 2022On the Convergence Rates of Policy Gradient Methodscited by 115
  8. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  9. 2022Discovered Policy Optimisationcited by 103
  10. 2020Mirror Descent Policy Optimizationcited by 102
  11. 2018Evolved Policy Gradientscited by 86
  12. 2020Discovering Reinforcement Learning Algorithmscited by 75
  13. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  14. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  15. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 44
  16. 2022evosax: JAX-based Evolution Strategiescited by 39
  17. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  18. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 29
  19. 2022Policy Optimization over General State and Action Spacescited by 21
  20. 2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 16
  21. 2022Adversarial Cheap Talkcited by 16
  22. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  23. 2017f-Divergence constrained policy improvementcited by 15
  24. 2021Actor-critic is implicitly biased towards high entropy optimal policiescited by 13
  25. 2023Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RLcited by 13
  26. 2023Policy Mirror Descent Inherently Explores Action Spacecited by 11
  27. 2023Decision-Aware Actor-Critic with Function Approximation and Theoretical Guaranteescited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025On the Effect of Regularization in Policy Mirror Descentcited by 1