Dual Approximation Policy Optimization
Not on the map. Why?
Abstract (arXiv)
We propose Dual Approximation Policy Optimization (DAPO), a framework that incorporates general function approximation into policy mirror descent methods. In contrast to the popular approach of using the L2-norm to measure function approximation errors, DAPO uses the dual Bregman divergence induced by the mirror map for policy projection. This duality framework has both theoretical and practical implications: not only does it achieve fast linear convergence with general function approximation, but it also includes several well-known practical methods as special cases, immediately providing strong convergence guarantees.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 2 Oct 2024.
Builds on
36 of 59 references.
References · 36
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2015Trust Region Policy Optimizationcited by 2,503
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
- 2019Provably Efficient Exploration in Policy Optimizationcited by 203
- 2017A unified view of entropy-regularized Markov decision processescited by 174
- 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
- 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
- 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
24 more
- 2022On the Convergence Rates of Policy Gradient Methodscited by 115
- 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
- 2020Mirror Descent Policy Optimizationcited by 102
- 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
- 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
- 2020Optimistic Policy Optimization with Bandit Feedbackcited by 79
- 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
- 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 51
- 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
- 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 44
- 2019Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policycited by 42
- 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
- 2022Mirror Descent with Relative Smoothness in Measure Spaces, with application to Sinkhorn and EMcited by 33
- 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
- 2022Target Network and Truncation Overcome The Deadly Triad in Q-Learningcited by 31
- 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 29
- 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
- 2022Policy Optimization over General State and Action Spacescited by 21
- 2023Optimal Convergence Rate for Exact Policy Mirror Descent in Discounted Markov Decision Processescited by 17
- 2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 16
- 2022Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexitycited by 14
- 2019Probability Functional Descent: A Unifying Perspective on GANs, Variational Inference, and Reinforcement Learningcited by 11
- 2023Policy Mirror Descent Inherently Explores Action Spacecited by 11
- 2022Block Policy Mirror Descentcited by 10
Built on by
Citing papers per year: 2025: 1.
1 paper.