mlsanityData to 7 Oct 2026

Off-Policy Policy Gradient with State Distribution Correction

arXiv 1904.08473 ↗First posted 17 Apr 2019cs.LG cs.AI stat.MLcited by 43

Not on the map. Why?

Abstract (arXiv)

We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method. Prior off-policy policy gradient approaches have generally ignored the mismatch between the distribution of states visited under the behavior policy used to collect data, and what would be the distribution of states under the learned policy. Here we build on recent progress for estimating the ratio of the state distributions under behavior and evaluation policies for policy evaluation, and present an off-policy policy gradient optimization technique that can account for this mismatch in distributions. We present an illustrative example of why this is important and a theoretical convergence guarantee for our approach. Empirically, we compare our method in simulations to several strong baselines which do not correct for this mismatch, significantly improving in the quality of the policy discovered.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 12; 2023: 8; 2024: 3; 2025: 2; 2026: 2.

43 papers.

Sort:

By area:

  1. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  2. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  3. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 53
  4. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  5. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  6. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  7. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  8. 2021Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approachcited by 27
  9. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  10. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
  11. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximationcited by 22
  12. 2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 19
31 more
  1. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendationcited by 18
  2. 2021Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatchcited by 17
  3. 2023B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesiscited by 16
  4. 2022Model-Based Offline Meta-Reinforcement Learning with Regularizationcited by 14
  5. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  6. 2023Reinforcement Learning Tutor Better Supported Lower Performers in a Math Taskcited by 11
  7. 2021Characterizing the Gap Between Actor-Critic and Policy Gradientcited by 9
  8. 2022Continual Auxiliary Task Learningcited by 9
  9. 2022Offline Policy Optimization with Eligible Actionscited by 7
  10. 2023A Policy Gradient Method for Confounded POMDPscited by 6
  11. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyondcited by 6
  12. 2021Learning Expected Emphatic Traces for Deep RLcited by 5
  13. 2021Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updatescited by 5
  14. 2021Off-Policy Actor-Critic with Emphatic Weightingscited by 5
  15. 2021On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correctioncited by 4
  16. 2022A Temporal-Difference Approach to Policy Gradient Estimationcited by 4
  17. 2023Budgeting Counterfactual for Offline RLcited by 4
  18. 2021The Curse of Passive Data Collection in Batch Reinforcement Learningcited by 2
  19. 2022Behavior Estimation from Multi-Source Data for Offline Reinforcement Learningcited by 2
  20. 2025On The Statistical Complexity of Offline Decision-Makingcited by 2
  21. 2021Neural Network Compatible Off-Policy Natural Actor-Critic Algorithmcited by 1
  22. 2022Optimal Estimation of Off-Policy Policy Gradient via Double Fitted Iterationcited by 1
  23. 2022Variance Reduction based Partial Trajectory Reuse to Accelerate Policy Gradient Optimizationcited by 1
  24. 2022Coordinate Ascent for Off-Policy RL with Global Convergence Guaranteescited by 1
  25. 2024Causal prompting model-based offline reinforcement learningcited by 1
  26. 2024SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasetscited by 1
  27. 2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
  28. 2026Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Modelscited by 1
  29. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learningcited by 1
  30. 2023Risk-Sensitive and Robust Model-Based Reinforcement Learning and Planningcited by 0
  31. 2023Reward Dropout Improves Control: Bi-objective Perspective on Reinforced LMcited by 0