mlsanityData to 7 Oct 2026

Off-OAB: Off-Policy Policy Gradient Method with Optimal Action-Dependent Baseline

arXiv 2405.02572 ↗First posted 4 May 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Policy-based methods have achieved remarkable success in solving challenging reinforcement learning problems. Among these methods, off-policy policy gradient methods are particularly important due to that they can benefit from off-policy data. However, these methods suffer from the high variance of the off-policy policy gradient (OPPG) estimator, which results in poor sample efficiency during training. In this paper, we propose an off-policy policy gradient method with the optimal action-dependent baseline (Off-OAB) to mitigate this variance issue. Specifically, this baseline maintains the OPPG estimator's unbiasedness while theoretically minimizing its variance. To enhance practical computational efficiency, we design an approximated version of this optimal baseline. Utilizing this approximation, our method (Off-OAB) aims to decrease the OPPG estimator's variance during policy optimization. We evaluate the proposed Off-OAB method on six representative tasks from OpenAI Gym and MuJoCo, where it demonstrably surpasses state-of-the-art methods on the majority of these tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 4 May 2024.

Builds on

34 of 61 references.

References · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2016OpenAI Gymcited by 1,684
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  11. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  12. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
22 more
  1. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  2. 2017Backpropagation through the Void: Optimizing control variates for black-box gradient estimationcited by 104
  3. 2018Stochastic Variance-Reduced Policy Gradientcited by 103
  4. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
  5. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  6. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  7. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  8. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
  9. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  10. 2017Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learningcited by 40
  11. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  12. 2021The ODE Method for Asymptotic Statistics in Stochastic Approximation and Reinforcement Learningcited by 29
  13. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  14. 2020Statistically Efficient Off-Policy Policy Gradientscited by 24
  15. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  16. 2018Clipped Action Policy Gradientcited by 17
  17. 2019Policy Optimization with Stochastic Mirror Descentcited by 17
  18. 2020Sample Complexity of Policy Gradient Finding Second-Order Stationary Pointscited by 16
  19. 2022RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on statescited by 16
  20. 2023Off-Policy Average Reward Actor-Critic with Deterministic Policy Searchcited by 10
  21. 2023Policy Gradient with Tree Expansioncited by 3
  22. 2023Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimizationcited by 3

Built on by

Not cited yet.