mlsanityData to 7 Oct 2026

An Off-policy Policy Gradient Theorem Using Emphatic Weightings

arXiv 1811.09013 ↗First posted 22 Nov 2018cs.LG stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

Policy gradient methods are widely used for control in reinforcement learning, particularly for the continuous action setting. There have been a host of theoretically sound algorithms proposed for the on-policy setting, due to the existence of the policy gradient theorem which provides a simplified form for the gradient. In off-policy learning, however, where the behaviour policy is not necessarily attempting to learn and follow the optimal policy for the given task, the existence of such a theorem has been elusive. In this work, we solve this open problem by providing the first off-policy policy gradient theorem. The key to the derivation is the use of emphatic weightings. We develop a new actor-critic algorithmunicodex2014called Actor Critic with Emphatic weightings (ACE)unicodex2014that approximates the simplified gradients provided by the theorem. We demonstrate in a simple counterexample that previous off-policy policy gradient methodsunicodex2014particularly OffPAC and DPGunicodex2014converge to the wrong solution whereas ACE finds the optimal solution.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 14; 2022: 4; 2023: 8; 2024: 5; 2025: 1; 2026: 1.

34 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2023Empirical Design in Reinforcement Learningcited by 54
  3. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  4. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
  5. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
  6. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  7. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 20
  8. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  9. 2023B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesiscited by 16
  10. 2021Reinforcement Learning Under Algorithmic Triagecited by 15
  11. 2021Emphatic Algorithms for Deep Reinforcement Learningcited by 12
  12. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
22 more
  1. 2021Truncated Emphatic Temporal Difference Methods for Prediction and Controlcited by 10
  2. 2022General Policy Evaluation and Improvement by Learning to Identify Few But Crucial Statescited by 10
  3. 2023Improving Offline RL by Blending Heuristicscited by 10
  4. 2021Model-free Policy Learning with Reward Gradientscited by 9
  5. 2023Correcting discount-factor mismatch in on-policy policy gradient methodscited by 9
  6. 2024Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Modelscited by 6
  7. 2021Learning Expected Emphatic Traces for Deep RLcited by 5
  8. 2021Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updatescited by 5
  9. 2021Off-Policy Actor-Critic with Emphatic Weightingscited by 5
  10. 2021On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correctioncited by 4
  11. 2022A Temporal-Difference Approach to Policy Gradient Estimationcited by 4
  12. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 4
  13. 2021PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Methodcited by 2
  14. 2024When Do Off-Policy and On-Policy Policy Gradient Methods Align?cited by 2
  15. 2021Neural Network Compatible Off-Policy Natural Actor-Critic Algorithmcited by 1
  16. 2022Coordinate Ascent for Off-Policy RL with Global Convergence Guaranteescited by 1
  17. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
  18. 2022Off-Policy Deep Reinforcement Learning Algorithms for Handling Various Robotic Manipulator Taskscited by 0
  19. 2023Reward Dropout Improves Control: Bi-objective Perspective on Reinforced LMcited by 0
  20. 2023Discerning Temporal Difference Learningcited by 0
  21. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improvecited by 0
  22. 2025Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Explorationcited by 0