mlsanityData to 7 Oct 2026

Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning

arXiv 1604.00923 ↗First posted 4 Apr 2016cs.LG cs.AIcited by 285

Not on the map. Why?

Cited by 62 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper we present a new way of predicting the performance of a reinforcement learning policy given historical data that may have been generated by a different policy. The ability to evaluate a policy from historical data is important for applications where the deployment of a bad policy can be dangerous or costly. We show empirically that our algorithm produces estimates that often have orders of magnitude lower mean squared error than existing methods---it makes more efficient use of the available data. Our new estimator is based on two advances: an extension of the doubly robust estimator (Jiang and Li, 2015), and a new way to mix between model based estimates and importance sampling based estimates.

Key facts. Cited by 285 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Apr 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 3; 2019: 4; 2020: 10; 2021: 57; 2022: 53; 2023: 49; 2024: 33; 2025: 37; 2026: 39.

285 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  4. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  5. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  6. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  7. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  8. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  9. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  10. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  11. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  12. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
38 more
  1. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  2. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  3. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 61
  4. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  5. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 53
  6. 2021Universal Off-Policy Evaluationcited by 43
  7. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  8. 2021Statistical Inference with M-Estimators on Adaptively Collected Datacited by 40
  9. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  10. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
  11. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  12. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  13. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  14. 2022Towards Data-Efficient Detection Transformerscited by 34
  15. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  16. 2021Recomposing the Reinforcement Learning Building Blocks with Hypernetworkscited by 33
  17. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedbackcited by 32
  18. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  19. 2021Post-Contextual-Bandit Inferencecited by 32
  20. 2022Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Modelcited by 30
  21. 2021Off-Policy Risk Assessment in Contextual Banditscited by 29
  22. 2021Recursive Causal Structure Learning in the Presence of Latent Variables and Selection Biascited by 29
  23. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 29
  24. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  25. 2021Fast Rates for the Regret of Offline Reinforcement Learningcited by 26
  26. 2022Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimationcited by 25
  27. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
  28. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  29. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  30. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  31. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPscited by 22
  32. 2022Markovian Interference in Experimentscited by 22
  33. 2022Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelinescited by 22
  34. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 20
  35. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  36. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  37. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  38. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19

50 most cited of 285 All 285 (JSON)