mlsanityData to 7 Oct 2026

Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updates

arXiv 2109.14727 ↗First posted 29 Sep 2021cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

The policy gradient theorem states that the policy should only be updated in states that are visited by the current policy, which leads to insufficient planning in the off-policy states, and thus to convergence to suboptimal policies. We tackle this planning issue by extending the policy gradient theory to policy updates with respect to any state density. Under these generalized policy updates, we show convergence to optimality under a necessary and sufficient condition on the updates' state densities, and thereby solve the aforementioned planning issue. We also prove asymptotic convergence rates that significantly improve those in the policy gradient literature. To implement the principles prescribed by our theory, we propose an agent, Dr Jekyll & Mr Hyde (JH), with a double personality: Dr Jekyll purely exploits while Mr Hyde purely explores. JH's independent policies allow to record two separate replay buffers: one on-policy (Dr Jekyll's) and one off-policy (Mr Hyde's), and therefore to update JH's models with a mixture of on-policy and off-policy updates. More than an algorithm, JH defines principles for actor-critic algorithms to satisfy the requirements we identify in our analysis. We extensively test on finite MDPs where JH demonstrates a superior ability to recover from converging to a suboptimal policy without impairing its speed of convergence. We also implement a deep version of the algorithm and test it on a simple problem where it shows promising results.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

30 of 52 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2018Exploration by Random Network Distillationcited by 966
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  10. 2016Deep Exploration via Bootstrapped DQNcited by 518
  11. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  12. 2018Distributed Prioritized Experience Replaycited by 260
18 more
  1. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  2. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  3. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  4. 2017Deep Exploration via Randomized Value Functionscited by 155
  5. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  6. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  7. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 51
  8. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  9. 2019Is the Policy Gradient a Gradient?cited by 40
  10. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  11. 2019rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorchcited by 38
  12. 2018An Off-policy Policy Gradient Theorem Using Emphatic Weightingscited by 34
  13. 2021Density Constrained Reinforcement Learningcited by 25
  14. 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
  15. 2019Generalized Off-Policy Actor-Criticcited by 16
  16. 2017Expected Policy Gradientscited by 14
  17. 2020A Deeper Look at Discounting Mismatch in Actor-Critic Algorithmscited by 10
  18. 2020OPAC: Opportunistic Actor-Criticcited by 2

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3.

5 papers.

  1. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  2. 2021Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatchcited by 17
  3. 2021Off-Policy Actor-Critic with Emphatic Weightingscited by 5
  4. 2022Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithmscited by 2
  5. 2022Non-Markovian policies occupancy measurescited by 0