# An Off-policy Policy Gradient Theorem Using Emphatic Weightings

arXiv: https://arxiv.org/abs/1811.09013 · HTML: https://mlsanity.com/paper/1811.09013/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2018.

## Built on by (most cited first, 30 of 34)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Empirical Design in Reinforcement Learning](https://mlsanity.com/paper/2304.01315/) (2023)
- [Dual RL: Unification and New Methods for Reinforcement and Imitation Learning](https://mlsanity.com/paper/2302.08560/) (2023)
- [Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences](https://mlsanity.com/paper/2107.08285/) (2021)
- [Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximation](https://mlsanity.com/paper/2105.12540/) (2021)
- [Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality](https://mlsanity.com/paper/2102.11866/) (2021)
- [Doubly Mild Generalization for Offline Reinforcement Learning](https://mlsanity.com/paper/2411.07934/) (2024)
- [A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation](https://mlsanity.com/paper/2106.06854/) (2021)
- [B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis](https://mlsanity.com/paper/2310.03173/) (2023)
- [Reinforcement Learning Under Algorithmic Triage](https://mlsanity.com/paper/2109.11328/) (2021)
- [Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders](https://mlsanity.com/paper/2302.00662/) (2023)
- [Emphatic Algorithms for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.11779/) (2021)
- [Truncated Emphatic Temporal Difference Methods for Prediction and Control](https://mlsanity.com/paper/2108.05338/) (2021)
- [General Policy Evaluation and Improvement by Learning to Identify Few But Crucial States](https://mlsanity.com/paper/2207.01566/) (2022)
- [Improving Offline RL by Blending Heuristics](https://mlsanity.com/paper/2306.00321/) (2023)
- [Model-free Policy Learning with Reward Gradients](https://mlsanity.com/paper/2103.05147/) (2021)
- [Correcting discount-factor mismatch in on-policy policy gradient methods](https://mlsanity.com/paper/2306.13284/) (2023)
- [Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models](https://mlsanity.com/paper/2402.03284/) (2024)
- [On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction](https://mlsanity.com/paper/2106.00993/) (2021)
- [Learning Expected Emphatic Traces for Deep RL](https://mlsanity.com/paper/2107.05405/) (2021)
- [Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updates](https://mlsanity.com/paper/2109.14727/) (2021)
- [Off-Policy Actor-Critic with Emphatic Weightings](https://mlsanity.com/paper/2111.08172/) (2021)
- [OMPO: A Unified Framework for RL under Policy and Dynamics Shifts](https://mlsanity.com/paper/2405.19080/) (2024)
- [A Temporal-Difference Approach to Policy Gradient Estimation](https://mlsanity.com/paper/2202.02396/) (2022)
- [PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Method](https://mlsanity.com/paper/2110.06906/) (2021)
- [When Do Off-Policy and On-Policy Policy Gradient Methods Align?](https://mlsanity.com/paper/2402.12034/) (2024)
- [Neural Network Compatible Off-Policy Natural Actor-Critic Algorithm](https://mlsanity.com/paper/2110.10017/) (2021)
- [Coordinate Ascent for Off-Policy RL with Global Convergence Guarantees](https://mlsanity.com/paper/2212.05237/) (2022)
- [Off-Policy Deep Reinforcement Learning Algorithms for Handling Various Robotic Manipulator Tasks](https://mlsanity.com/paper/2212.05572/) (2022)
- [Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2602.12375/) (2026)

## Abstract (arXiv)

Policy gradient methods are widely used for control in reinforcement learning, particularly for the continuous action setting. There have been a host of theoretically sound algorithms proposed for the on-policy setting, due to the existence of the policy gradient theorem which provides a simplified form for the gradient. In off-policy learning, however, where the behaviour policy is not necessarily attempting to learn and follow the optimal policy for the given task, the existence of such a theorem has been elusive. In this work, we solve this open problem by providing the first off-policy policy gradient theorem. The key to the derivation is the use of emphatic weightings. We develop a new actor-critic algorithmunicodex2014called Actor Critic with Emphatic weightings (ACE)unicodex2014that approximates the simplified gradients provided by the theorem. We demonstrate in a simple counterexample that previous off-policy policy gradient methodsunicodex2014particularly OffPAC and DPGunicodex2014converge to the wrong solution whereas ACE finds the optimal solution.