mlsanityData to 7 Oct 2026

Offline Policy Comparison with Confidence: Benchmarks and Baselines

arXiv 2205.10739 ↗First posted 22 May 2022cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Decision makers often wish to use offline historical data to compare sequential-action policies at various world states. Importantly, computational tools should produce confidence values for such offline policy comparison (OPC) to account for statistical variance and limited data coverage. Nevertheless, there is little work that directly evaluates the quality of confidence values for OPC. In this work, we address this issue by creating benchmarks for OPC with Confidence (OPCC), derived by adding sets of policy comparison queries to datasets from offline reinforcement learning. In addition, we present an empirical evaluation of the risk versus coverage trade-off for a class of model-based baselines. In particular, the baselines learn ensembles of dynamics models, which are used in various ways to produce simulations for answering queries with confidence values. While our results suggest advantages for certain baseline variations, there appears to be significant room for improvement in future work.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 93 references that are in the map. First posted on arXiv 22 May 2022.

Builds on

49 of 93 references.

References · 49

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2016OpenAI Gymcited by 1,684
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2018World Modelscited by 1,444
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
37 more
  1. 2017Selective Classification for Deep Neural Networkscited by 799
  2. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  5. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  6. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  8. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  9. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  10. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  11. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  12. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  13. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  14. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  15. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  16. 2020Model-Based Offline Planningcited by 126
  17. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  18. 2021Machine Learning with a Reject Option: A surveycited by 107
  19. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  20. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  21. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  22. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  23. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  24. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  25. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  26. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  27. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  28. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  29. 2020Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weightingcited by 38
  30. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  31. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  32. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  33. 2015Performance measures for classification systems with rejectioncited by 16
  34. 2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPscited by 16
  35. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14
  36. 2020Dream and Search to Control: Latent Space Planning for Continuous Controlcited by 5
  37. 2021Supervised Off-Policy Rankingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  2. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  3. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Data
  4. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators
  5. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  6. 2021Active Offline Policy Selection
  7. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  8. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data