# Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning

arXiv: https://arxiv.org/abs/1906.03735 · HTML: https://mlsanity.com/paper/1906.03735/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2019.

## Built on by (most cited first, 28 of 28)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Post-Contextual-Bandit Inference](https://mlsanity.com/paper/2106.00418/) (2021)
- [Evaluating the Robustness of Off-Policy Evaluation](https://mlsanity.com/paper/2108.13703/) (2021)
- [Policy-Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2211.13904/) (2022)
- [Model-agnostic meta-learners for estimating heterogeneous treatment effects over time](https://mlsanity.com/paper/2407.05287/) (2024)
- [Off-policy Confidence Sequences](https://mlsanity.com/paper/2102.09540/) (2021)
- [Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings](https://mlsanity.com/paper/2105.06029/) (2021)
- [Control Variates for Slate Off-Policy Evaluation](https://mlsanity.com/paper/2106.07914/) (2021)
- [Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation](https://mlsanity.com/paper/2311.18207/) (2023)
- [Risk Minimization from Adaptively Collected Data: Guarantees for Supervised and Policy Learning](https://mlsanity.com/paper/2106.01723/) (2021)
- [A Survey of Deep Causal Models and Their Industrial Applications](https://mlsanity.com/paper/2209.08860/) (2022)
- [IGC-Net for conditional average potential outcome estimation over time](https://mlsanity.com/paper/2405.21012/) (2024)
- [Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning](https://mlsanity.com/paper/2006.03886/) (2020)
- [Treatment Effect Estimation for Optimal Decision-Making](https://mlsanity.com/paper/2505.13092/) (2025)
- [Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Framework](https://mlsanity.com/paper/2309.13278/) (2023)
- [SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluation](https://mlsanity.com/paper/2311.18206/) (2023)
- [Winner's Curse Drives False Promises in Data-Driven Decisions: A Case Study in Refugee Matching](https://mlsanity.com/paper/2602.08892/) (2026)
- [Distributional Off-Policy Evaluation for Slate Recommendations](https://mlsanity.com/paper/2308.14165/) (2023)
- [Automated Off-Policy Estimator Selection via Supervised Learning](https://mlsanity.com/paper/2406.18022/) (2024)
- [Clustering Context in Off-Policy Evaluation](https://mlsanity.com/paper/2502.21304/) (2025)
- [Bayesian Inference of Contextual Bandit Policies via Empirical Likelihood](https://mlsanity.com/paper/2602.10608/) (2026)
- [Asymptotically Unbiased Off-Policy Policy Evaluation when Reusing Old Data in Nonstationary Environments](https://mlsanity.com/paper/2302.11725/) (2023)
- [Adaptive Doubly Robust Estimator from Non-stationary Logging Policy under a Convergence of Average Probability](https://mlsanity.com/paper/2102.08975/) (2021)
- [A Causal Approach for Business Optimization: Application on an Online Marketplace](https://mlsanity.com/paper/2207.01722/) (2022)
- [IntOPE: Off-Policy Evaluation in the Presence of Interference](https://mlsanity.com/paper/2408.13484/) (2024)
- [Kernel weighted importance sampling for off-policy evaluation in contextual bandits](https://mlsanity.com/paper/2607.15067/) (2026)
- [Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning](https://mlsanity.com/paper/2608.24858/) (2026)

## Abstract (arXiv)

Off-policy evaluation (OPE) in both contextual bandits and reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible. The problem's importance has attracted many proposed solutions, including importance sampling (IS), self-normalized IS (SNIS), and doubly robust (DR) estimates. DR and its variants ensure semiparametric local efficiency if Q-functions are well-specified, but if they are not they can be worse than both IS and SNIS. It also does not enjoy SNIS's inherent stability and boundedness. We propose new estimators for OPE based on empirical likelihood that are always more efficient than IS, SNIS, and DR and satisfy the same stability and boundedness properties as SNIS. On the way, we categorize various properties and classify existing estimators by them. Besides the theoretical guarantees, empirical studies suggest the new estimators provide advantages.