mlsanityData to 14 Oct 2026

Functional Acceleration for Policy Mirror Descent

arXiv 2407.16602 ↗First posted 23 Jul 2024cs.LG cs.AI stat.MLcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We apply functional acceleration to the Policy Mirror Descent (PMD) general family of algorithms, which cover a wide range of novel and fundamental methods in Reinforcement Learning (RL). Leveraging duality, we propose a momentum-based PMD update. By taking the functional route, our approach is independent of the policy parametrization and applicable to large-scale optimization, covering previous applications of momentum at the level of policy parameters as a special case. We theoretically analyze several properties of this approach and complement with a numerical ablation study, which serves to illustrate the policy optimization dynamics on the value polytope, relative to different algorithmic design choices in this space. We further characterize numerically several features of the problem setting relevant for functional acceleration, and lastly, we investigate the impact of approximation on their learning mechanics.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 37 of its 75 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

37 of 75 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Policy Mirror Descent with Lookahead21 Mar

Other references · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  10. 2014Convex Optimization: Algorithms and Complexitycited by 575
  11. 2018Maximum a Posteriori Policy Optimisationcited by 328
  12. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
24 more
  1. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  2. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  3. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  4. 2016Relatively-Smooth Convex Optimization by First-Order Methods, and Applicationscited by 108
  5. 2020Mirror Descent Policy Optimizationcited by 106
  6. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 52
  7. 2019Learning When-to-Treat Policiescited by 49
  8. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  9. 2018A Forward-Backward Splitting Method for Monotone Inclusions Without Cocoercivitycited by 36
  10. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 30
  11. 2019The Value Function Polytope in Reinforcement Learningcited by 29
  12. 2018Acceleration through Optimistic No-Regret Dynamicscited by 22
  13. 2021No-Regret Dynamics in the Fenchel Game: A Unified Framework for Algorithmic Convex Optimizationcited by 20
  14. 2019Momentum in Reinforcement Learningcited by 19
  15. 2021Understanding the Effect of Stochasticity in Policy Optimizationcited by 18
  16. 2023Optimal Convergence Rate for Exact Policy Mirror Descent in Discounted Markov Decision Processescited by 18
  17. 2019A First-Order Approach To Accelerated Value Iterationcited by 17
  18. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  19. 2018Predictor-Corrector Policy Optimizationcited by 9
  20. 2021From Convex Optimization to MDPs: A Review of First-Order, Second-Order and Quasi-Newton Methods for MDPscited by 9
  21. 2020Two steps at a time -- taking GAN training in stride with Tseng's methodcited by 8
  22. 2015Accelerating Optimization via Adaptive Predictioncited by 5
  23. 2020Approximation Benefits of Policy Gradient Methods with Aggregated Statescited by 5
  24. 2023Decision-Aware Actor-Critic with Function Approximation and Theoretical Guaranteescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025On the Convergence of Policy Mirror Descent with Temporal Difference Evaluationcited by 3
  2. 2025Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Datacited by 2
  3. 2026Fast Regularized Policy Mirror Descent with One-Step TD Updatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent
  2. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
  3. 2025On the Effect of Regularization in Policy Mirror Descent
  4. 2024Learning mirror maps in policy mirror descent
  5. 2026Approximation of Log-Partition Function in Policy Mirror Descent Induces Implicit Regularization for LLM Post-Training
  6. 2026Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity
  7. 2024Operator World Models for Reinforcement Learning
  8. 2026Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces