mlsanityData to 7 Oct 2026

Benchmarks for Deep Off-Policy Evaluation

arXiv 2103.16596 ↗First posted 30 Mar 2021cs.LG stat.MLcited by 89

Not on the map. Why?

Cited by 36 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both evaluating and selecting complex policies for decision making. The ability to learn offline is particularly important in many real-world domains, such as in healthcare, recommender systems, or robotics, where online data collection is an expensive and potentially dangerous process. Being able to accurately evaluate and select high-performing policies without requiring online interaction could yield significant benefits in safety, time, and cost for these applications. While many OPE methods have been proposed in recent years, comparing results between papers is difficult because currently there is a lack of a comprehensive and unified benchmark, and measuring algorithmic progress has been challenging due to the lack of difficult evaluation tasks. In order to address this gap, we present a collection of policies that in conjunction with existing offline datasets can be used for benchmarking off-policy evaluation. Our tasks include a range of challenging high-dimensional continuous control problems, with wide selections of datasets and policies for performing policy selection. The goal of our benchmark is to provide a standardized measure of progress that is motivated from a set of principles designed to challenge and test the limits of existing OPE methods. We perform an evaluation of state-of-the-art algorithms and provide open-source access to our data and code to foster future research in this area.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

37 of 62 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  12. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
25 more
  1. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  2. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  3. 2020Critic Regularized Regressioncited by 269
  4. 2018Learning to Drive in a Daycited by 266
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2019Batch Policy Learning under Constraintscited by 242
  8. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  9. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  10. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  11. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  12. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  13. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  14. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  15. 2016Off-policy evaluation for slate recommendationcited by 102
  16. 2018Offline A/B testing for Recommender Systemscited by 94
  17. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  18. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  19. 2019Learning When-to-Treat Policiescited by 49
  20. 2018Importance Sampling Policy Evaluation with an Estimated Behavior Policycited by 34
  21. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  22. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  23. 2020Batch Stationary Distribution Estimationcited by 19
  24. 2018Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matterscited by 13
  25. 2018Off-Policy Evaluation and Learning from Logged Bandit Feedback: Error Reduction via Surrogate Policycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 21; 2023: 22; 2024: 10; 2025: 9; 2026: 8.

89 papers.

Sort:

By area:

  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  4. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  5. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  6. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  7. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  8. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  9. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  10. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  11. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  12. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
38 more
  1. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  2. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 61
  3. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrationscited by 51
  4. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  5. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  6. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  7. 2021RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learningcited by 31
  8. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
  9. 2024Improving Agent Behaviors with RL Fine-tuning for Autonomous Drivingcited by 27
  10. 2021Active Offline Policy Selectioncited by 26
  11. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  12. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  13. 2022Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetescited by 22
  14. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  15. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  16. 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
  17. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  18. 2023Asking for Help: Failure Prediction in Behavioral Cloning through Value Approximationcited by 15
  19. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraintscited by 13
  20. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  21. 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
  22. 2022Adversarial Counterfactual Environment Model Learningcited by 12
  23. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  24. 2023A Survey of Demonstration Learningcited by 12
  25. 2022User-Interactive Offline Reinforcement Learningcited by 11
  26. 2023Active Policy Improvement from Multiple Black-box Oraclescited by 11
  27. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learningcited by 9
  28. 2023Offline Learning of Closed-Loop Deep Brain Stimulation Controllers for Parkinson Disease Treatmentcited by 8
  29. 2022Off-Policy Evaluation with Online Adaptation for Robot Exploration in Challenging Environmentscited by 7
  30. 2022Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstractioncited by 7
  31. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  32. 2023Off-Policy Evaluation for Human Feedbackcited by 6
  33. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  34. 2021SOPE: Spectrum of Off-Policy Estimatorscited by 5
  35. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 5
  36. 2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
  37. 2023HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcarecited by 5
  38. 2023Marginalized Importance Sampling for Off-Environment Policy Evaluationcited by 5

50 most cited of 89 All 89 (JSON)