# OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators

arXiv: https://arxiv.org/abs/2405.17708 · HTML: https://mlsanity.com/paper/2405.17708/

## Key facts

- Cited by 3 papers in the map (data through 7 Oct 2026).
- Builds on 36 of its 63 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 30 of 36)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections](https://mlsanity.com/paper/1906.04733/) (2019)
- [Batch Policy Learning under Constraints](https://mlsanity.com/paper/1903.08738/) (2019)
- [Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation](https://mlsanity.com/paper/1810.12429/) (2018)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1911.06854/) (2019)
- [Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models](https://mlsanity.com/paper/1905.05824/) (2019)
- [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016)
- [Batch Value-function Approximation with Only Realizability](https://mlsanity.com/paper/2008.04990/) (2020)
- [Benchmarks for Deep Off-Policy Evaluation](https://mlsanity.com/paper/2103.16596/) (2021)
- [Off-Policy Evaluation via the Regularized Lagrangian](https://mlsanity.com/paper/2007.03438/) (2020)
- [Off-Policy Evaluation in Partially Observable Environments](https://mlsanity.com/paper/1909.03739/) (2019)
- [Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding](https://mlsanity.com/paper/2003.05623/) (2020)
- [Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning](https://mlsanity.com/paper/2002.04518/) (2020)
- [Double/Debiased Machine Learning for Treatment and Causal Parameters](https://mlsanity.com/paper/1608.00060/) (2016)
- [Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2002.07729/) (2020)
- [A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2111.06784/) (2021)
- [Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning](https://mlsanity.com/paper/2110.14000/) (2021)
- [Representation Balancing MDPs for Off-Policy Policy Evaluation](https://mlsanity.com/paper/1805.09044/) (2018)
- [A Spectral Approach to Off-Policy Evaluation for POMDPs](https://mlsanity.com/paper/2109.10502/) (2021)
- [Combining Parametric and Nonparametric Models for Off-Policy Evaluation](https://mlsanity.com/paper/1905.05787/) (2019)
- [Learning Bellman Complete Representations for Offline Policy Evaluation](https://mlsanity.com/paper/2207.05837/) (2022)

## Built on by (most cited first, 3 of 3)

- [PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data](https://mlsanity.com/paper/2507.20068/) (2025)
- [MORE-CLEAR: Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced State Representation](https://mlsanity.com/paper/2508.07681/) (2025)
- [Meta Off-Policy Estimation](https://mlsanity.com/paper/2508.07914/) (2025)

## Abstract (arXiv)

Offline policy evaluation (OPE) allows us to evaluate and estimate a new sequential decision-making policy's performance by leveraging historical interaction data collected from other policies. Evaluating a new policy online without a confident estimate of its performance can lead to costly, unsafe, or hazardous outcomes, especially in education and healthcare. Several OPE estimators have been proposed in the last decade, many of which have hyperparameters and require training. Unfortunately, choosing the best OPE algorithm for each task and domain is still unclear. In this paper, we propose a new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure. We prove that our estimator is consistent and satisfies several desirable properties for policy evaluation. Additionally, we demonstrate that when compared to alternative approaches, our estimator can be used to select higher-performing policies in healthcare and robotics. Our work contributes to improving ease of use for a general-purpose, estimator-agnostic, off-policy evaluation framework for offline RL.