mlsanityData to 7 Oct 2026

Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces

arXiv 2603.10199 ↗First posted 10 Mar 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Policy Dual Averaging (PDA) offers a principled Policy Mirror Descent (PMD) framework that more naturally admits value function approximation than standard PMD, enabling the use of approximate advantage (or Q-) functions while retaining strong convergence guarantees. However, applying PDA in continuous state and action spaces remains computationally challenging, since action selection involves solving an optimization sub-problem at each decision step. In this paper, we propose textitactor-accelerated PDA, which uses a learned policy network to approximate the solution of the optimization sub-problems, yielding faster runtimes while maintaining convergence guarantees. We provide a theoretical analysis that quantifies how actor approximation error impacts the convergence of PDA under suitable assumptions. We then evaluate its performance on several benchmarks in robotics, control, and operations research problems. Actor-accelerated PDA achieves superior performance compared to popular on-policy baselines such as Proximal Policy Optimization (PPO). Overall, our results bridge the gap between the theoretical advantages of PDA and its practical deployment in continuous-action problems with function approximation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

21 of 36 references.

References · 21

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  6. 2017Deep Reinforcement Learning that Matterscited by 727
  7. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  8. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  9. 2017A unified view of entropy-regularized Markov decision processescited by 174
  10. 2018Stochastic model-based minimization of weakly convex functionscited by 157
  11. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  12. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
9 more
  1. 2020Mirror Descent Policy Optimizationcited by 102
  2. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
  3. 2020OR-Gym: A Reinforcement Learning Library for Operations Research Problemscited by 50
  4. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 29
  5. 2019ORL: Reinforcement Learning Benchmarks for Online Stochastic Optimization Problemscited by 21
  6. 2022Policy Optimization over General State and Action Spacescited by 21
  7. 2023Policy Mirror Descent Inherently Explores Action Spacecited by 11
  8. 2022Block Policy Mirror Descentcited by 10
  9. 2024Strongly-polynomial time and validation analysis of policy gradient methodscited by 5

Built on by

Not cited yet.