mlsanityData to 13 Oct 2026

From Importance Sampling to Doubly Robust Policy Gradient

arXiv 1910.09066 ↗First posted 20 Oct 2019cs.LG stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

We show that on-policy policy gradient (PG) and its variance reduction variants can be derived by taking finite difference of function evaluations supplied by estimators from the importance sampling (IS) family for off-policy evaluation (OPE). Starting from the doubly robust (DR) estimator (Jiang & Li, 2016), we provide a simple derivation of a very general and flexible form of PG, which subsumes the state-of-the-art variance reduction technique (Cheng et al., 2019) as its special case and immediately hints at further variance reduction opportunities overlooked by existing literature. We analyze the variance of the new DR-PG estimator, compare it to existing methods as well as the Cramer-Rao lower bound of policy gradient, and empirically show its effectiveness.

Key facts. Cited by 17 papers in the map (data through 13 Oct 2026). First posted on arXiv 20 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2020: 1; 2021: 6; 2022: 2; 2023: 2; 2024: 1; 2025: 2; 2026: 3.

17 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,889
  2. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 20
  3. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learningcited by 18
  4. 2021Adaptive Importance Sampling meets Mirror Descent: a Bias-variance tradeoffcited by 13
  5. 2022Offline Policy Optimization with Eligible Actionscited by 7
  6. 2022Off-Policy Risk Assessment in Markov Decision Processescited by 7
  7. 2021Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluationcited by 5
  8. 2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimationcited by 2
  9. 2021A Unified Off-Policy Evaluation Approach for General Value Functioncited by 1
  10. 2021Coordinate-wise Control Variates for Deep Policy Gradientscited by 1
  11. 2023Decision-Focused Evaluation: Analyzing Performance of Deployed Restless Multi-Arm Banditscited by 1
  12. 2023Recent Advances in the Foundations and Applications of Unbiased Learning to Rankcited by 1
5 more
  1. 2025Seldonian Reinforcement Learning for Ad Hoc Teamworkcited by 1
  2. 2024Improving Reward-Conditioned Policies for Multi-Armed Bandits using Normalized Weight Functionscited by 0
  3. 2026Off-Policy Evaluation and Learning for Survival Outcomes under Censoringcited by 0
  4. 2026Policy Gradient Steering: Interventions from Behavioral Objectivescited by 0
  5. 2026When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPOcited by 0