mlsanityData to 14 Oct 2026

Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

arXiv 1511.03722 ↗First posted 11 Nov 2015cs.LG cs.AI cs.SYcited by 339

Not on the map. Why?

Cited by 71 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study the problem of off-policy value evaluation in reinforcement learning (RL), where one aims to estimate the value of a new policy based on data collected by a different policy. This problem is often a critical step when applying RL in real-world problems. Despite its importance, existing general methods either have uncontrolled bias or suffer high variance. In this work, we extend the doubly robust estimator for bandits to sequential decision-making problems, which gets the best of both worlds: it is guaranteed to be unbiased and can have a much lower variance than the popular importance sampling estimators. We demonstrate the estimator's accuracy in several benchmark problems, and illustrate its use as a subroutine in safe policy improvement. We also provide theoretical results on the hardness of the problem, and show that our estimator can match the lower bound in certain scenarios.

Key facts. Cited by 339 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 3; 2019: 3; 2020: 10; 2021: 72; 2022: 51; 2023: 54; 2024: 45; 2025: 43; 2026: 58.

339 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  3. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  6. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  7. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  8. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  9. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  10. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  11. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  12. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
38 more
  1. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  2. 2021Conservative Offline Distributional Reinforcement Learningcited by 79
  3. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  4. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 69
  5. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  6. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  7. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 64
  8. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  9. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  10. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56
  11. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 56
  12. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 54
  13. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52
  14. 2021Universal Off-Policy Evaluationcited by 43
  15. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  16. 2021Understanding the role of importance weighting for deep learningcited by 39
  17. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  18. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)cited by 38
  19. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
  20. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  21. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  22. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  23. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  24. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  25. 2026Your Group-Relative Advantage Is Biasedcited by 34
  26. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedbackcited by 32
  27. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
  28. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  29. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 31
  30. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  31. 2022Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Modelcited by 30
  32. 2021Off-Policy Risk Assessment in Contextual Banditscited by 29
  33. 2021Recursive Causal Structure Learning in the Presence of Latent Variables and Selection Biascited by 29
  34. 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
  35. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  36. 2021Active Offline Policy Selectioncited by 26
  37. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  38. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26

50 most cited of 339 All 339 (JSON)