mlsanityData to 7 Oct 2026

Mirror Descent Policy Optimization

arXiv 2005.09814 ↗First posted 20 May 2020cs.LG cs.AI stat.MLcited by 102

Not on the map. Why?

Abstract (arXiv)

Mirror descent (MD), a well-known first-order method in constrained convex optimization, has recently been shown as an important tool to analyze trust-region algorithms in reinforcement learning (RL). However, there remains a considerable gap between such theoretically analyzed algorithms and the ones used in practice. Inspired by this, we propose an efficient RL algorithm, called em mirror descent policy optimization (MDPO). MDPO iteratively updates the policy by em approximately solving a trust-region problem, whose objective function consists of two terms: a linearization of the standard RL objective and a proximity term that restricts two consecutive policies to be close to each other. Each update performs this approximation by taking multiple gradient steps on this objective function. We derive em on-policy and em off-policy variants of MDPO, while emphasizing important design choices motivated by the existing theory of MD in RL. We highlight the connections between on-policy MDPO and two popular trust-region RL algorithms: TRPO and PPO, and show that explicitly enforcing the trust-region constraint is in fact em not a necessity for high performance gains in TRPO. We then show how the popular soft actor-critic (SAC) algorithm can be derived by slight modifications of off-policy MDPO. Overall, MDPO is derived from the MD principles, offers a unified approach to viewing a number of popular RL algorithms, and performs better than or on-par with TRPO, PPO, and SAC in a number of continuous control tasks. Code is available at urlhttps://github.com/manantomar/Mirror-Descent-Policy-Optimization.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 11; 2023: 11; 2024: 15; 2025: 20; 2026: 28.

102 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  2. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  3. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  4. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  5. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 108
  6. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
  7. 2021Reward is enough for convex MDPscited by 82
  8. 2021A general sample complexity analysis of vanilla policy gradientcited by 74
  9. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  10. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  11. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 64
  12. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
38 more
  1. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  2. 2021Bootstrapped Meta-Learningcited by 45
  3. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 44
  4. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  5. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  6. 2021Batch size-invariance for policy optimizationcited by 33
  7. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  8. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 29
  9. 2021Online Apprenticeship Learningcited by 28
  10. 2022Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Gamescited by 24
  11. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  12. 2022You May Not Need Ratio Clipping in PPOcited by 19
  13. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 19
  14. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 19
  15. 2021Improved Regret Bound and Experience Replay in Regularized Policy Iterationcited by 18
  16. 2025One-Step Flow Policy Mirror Descentcited by 18
  17. 2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 16
  18. 2021Bregman Gradient Policy Optimizationcited by 15
  19. 2023Vision Transformer Adapters for Generalizable Multitask Learningcited by 14
  20. 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
  21. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
  22. 2022On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spacescited by 11
  23. 2021Optimization Issues in KL-Constrained Approximate Policy Iterationcited by 10
  24. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 10
  25. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 9
  26. 2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 9
  27. 2021Faster Deep Reinforcement Learning with Slower Online Networkcited by 7
  28. 2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spacescited by 7
  29. 2024Policy Mirror Descent with Lookaheadcited by 6
  30. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  31. 2026QuRL: Efficient Reinforcement Learning with Quantized Rolloutcited by 6
  32. 2021Dealing with Non-Stationarity in MARL via Trust-Region Decompositioncited by 5
  33. 2022Robust Imitation via Mirror Descent Inverse Reinforcement Learningcited by 5
  34. 2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
  35. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMscited by 5
  36. 2026Rethinking the Divergence Regularization in LLM RLcited by 5
  37. 2022Learning to Constrain Policy Optimization with Virtual Trust Regioncited by 4
  38. 2024Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reductioncited by 4

50 most cited of 102 All 102 (JSON)