mlsanityData to 14 Oct 2026

Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning

arXiv 1911.06854 ↗First posted 15 Nov 2019cs.LG cs.AI cs.ROcited by 121

Not on the map. Why?

Cited by 38 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We offer an experimental benchmark and empirical study for off-policy policy evaluation (OPE) in reinforcement learning, which is a key problem in many safety critical applications. Given the increasing interest in deploying learning-based methods, there has been a flurry of recent proposals for OPE method, leading to a need for standardized empirical analyses. Our work takes a strong focus on diversity of experimental design to enable stress testing of OPE methods. We provide a comprehensive benchmarking suite to study the interplay of different attributes on method performance. We distill the results into a summarized set of guidelines for OPE in practice. Our software package, the Caltech OPE Benchmarking Suite (COBS), is open-sourced and we invite interested researchers to further contribute to the benchmark.

Key facts. Cited by 121 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 2; 2021: 27; 2022: 23; 2023: 23; 2024: 18; 2025: 13; 2026: 15.

121 papers.

Sort:

By area:

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  2. 2021Offline RL Without Off-Policy Evaluationcited by 151
  3. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  4. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  5. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  6. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 69
  7. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  8. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  9. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  10. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 54
  11. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 54
  12. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
38 more
  1. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  2. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
  3. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
  4. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  5. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 32
  6. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 29
  7. 2022Performative Reinforcement Learningcited by 27
  8. 2021Active Offline Policy Selectioncited by 26
  9. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  10. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  11. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  12. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  13. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  14. 2022Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systemscited by 22
  15. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  16. 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
  17. 2021Minimax Model Learningcited by 16
  18. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  19. 2022ReVar: Strengthening Policy Evaluation via Reduced Variance Samplingcited by 16
  20. 2023Asking for Help: Failure Prediction in Behavioral Cloning through Value Approximationcited by 16
  21. 2021On Blame Attribution for Accountable Multi-Agent Sequential Decision Makingcited by 13
  22. 2021Safe Autonomous Racing via Approximate Reachability on Ego-visioncited by 13
  23. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  24. 2024POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decompositioncited by 13
  25. 2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
  26. 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
  27. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  28. 2023A Survey of Demonstration Learningcited by 12
  29. 2023Marginal Density Ratio for Off-Policy Evaluation in Contextual Banditscited by 9
  30. 2024Long-term Off-Policy Evaluation and Learningcited by 9
  31. 2021Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networkscited by 8
  32. 2022Efficient Offline Policy Optimization with a Learned Modelcited by 8
  33. 2022Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributionscited by 8
  34. 2024DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regimecited by 8
  35. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPscited by 8
  36. 2022Off-Policy Evaluation with Online Adaptation for Robot Exploration in Challenging Environmentscited by 7
  37. 2022Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstractioncited by 7
  38. 2023Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluationcited by 7

50 most cited of 121 All 121 (JSON)