mlsanityData to 7 Oct 2026

OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators

arXiv 2405.17708 ↗First posted 27 May 2024cs.LG cs.AI stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Offline policy evaluation (OPE) allows us to evaluate and estimate a new sequential decision-making policy's performance by leveraging historical interaction data collected from other policies. Evaluating a new policy online without a confident estimate of its performance can lead to costly, unsafe, or hazardous outcomes, especially in education and healthcare. Several OPE estimators have been proposed in the last decade, many of which have hyperparameters and require training. Unfortunately, choosing the best OPE algorithm for each task and domain is still unclear. In this paper, we propose a new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure. We prove that our estimator is consistent and satisfies several desirable properties for policy evaluation. Additionally, we demonstrate that when compared to alternative approaches, our estimator can be used to select higher-performing policies in healthcare and robotics. Our work contributes to improving ease of use for a general-purpose, estimator-agnostic, off-policy evaluation framework for offline RL.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 63 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

36 of 63 references.

References · 36

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  8. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 328
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2019Batch Policy Learning under Constraintscited by 242
  12. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
24 more
  1. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  2. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  3. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
  4. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  5. 2020Batch Value-function Approximation with Only Realizabilitycited by 103
  6. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  7. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  8. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  9. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  10. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  11. 2016Double/Debiased Machine Learning for Treatment and Causal Parameterscited by 41
  12. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
  13. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  14. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  15. 2018Representation Balancing MDPs for Off-Policy Policy Evaluationcited by 20
  16. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 20
  17. 2019Combining Parametric and Nonparametric Models for Off-Policy Evaluationcited by 17
  18. 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
  19. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learningcited by 13
  20. 2023Reinforcement Learning Tutor Better Supported Lower Performers in a Math Taskcited by 11
  21. 2024Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlapcited by 7
  22. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  23. 2021SOPE: Spectrum of Off-Policy Estimatorscited by 5
  24. 2023Model-based Offline Reinforcement Learning with Local Misspecificationcited by 5

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Datacited by 3
  2. 2025MORE-CLEAR: Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced State Representationcited by 2
  3. 2025Meta Off-Policy Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Active Offline Policy Selection
  2. 2023Off-Policy Evaluation for Human Feedback
  3. 2025GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents
  4. 2023When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspective
  5. 2024CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation
  6. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
  7. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  8. 2022Offline Policy Evaluation and Optimization under Confounding