# Adaptive Estimator Selection for Off-Policy Evaluation

arXiv: https://arxiv.org/abs/2002.07729 · HTML: https://mlsanity.com/paper/2002.07729/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2020.

## Built on by (most cited first, 30 of 38)

- [Off-Policy Evaluation for Large Action Spaces via Embeddings](https://mlsanity.com/paper/2202.06317/) (2022)
- [Deeply-Debiased Off-Policy Interval Estimation](https://mlsanity.com/paper/2105.04646/) (2021)
- [Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling](https://mlsanity.com/paper/2305.08062/) (2023)
- [Evaluating the Robustness of Off-Policy Evaluation](https://mlsanity.com/paper/2108.13703/) (2021)
- [Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorability](https://mlsanity.com/paper/2110.12343/) (2021)
- [Policy-Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2211.13904/) (2022)
- [Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstraction](https://mlsanity.com/paper/2402.02171/) (2024)
- [Off-Policy Evaluation for Large Action Spaces via Policy Convolution](https://mlsanity.com/paper/2310.15433/) (2023)
- [Oracle Inequalities for Model Selection in Offline Reinforcement Learning](https://mlsanity.com/paper/2211.02016/) (2022)
- [On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation](https://mlsanity.com/paper/2307.15053/) (2023)
- [Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation](https://mlsanity.com/paper/2311.18207/) (2023)
- [POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decomposition](https://mlsanity.com/paper/2402.06151/) (2024)
- [Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data](https://mlsanity.com/paper/2210.08642/) (2022)
- [Model Selection in Batch Policy Optimization](https://mlsanity.com/paper/2112.12320/) (2021)
- [Effective Off-Policy Evaluation and Learning in Contextual Combinatorial Bandits](https://mlsanity.com/paper/2408.11202/) (2024)
- [Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actions](https://mlsanity.com/paper/2210.13373/) (2022)
- [SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluation](https://mlsanity.com/paper/2311.18206/) (2023)
- [Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol](https://mlsanity.com/paper/2502.08021/) (2025)
- [When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspective](https://mlsanity.com/paper/2311.14110/) (2023)
- [Pessimistic Model Selection for Offline Deep Reinforcement Learning](https://mlsanity.com/paper/2111.14346/) (2021)
- [When is Offline Policy Selection Sample Efficient for Reinforcement Learning?](https://mlsanity.com/paper/2312.02355/) (2023)
- [Cross-Validated Off-Policy Evaluation](https://mlsanity.com/paper/2405.15332/) (2024)
- [OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators](https://mlsanity.com/paper/2405.17708/) (2024)
- [Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies](https://mlsanity.com/paper/2405.18792/) (2024)
- [Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation](https://mlsanity.com/paper/2410.02172/) (2024)
- [Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning](https://mlsanity.com/paper/2106.00589/) (2021)
- [Double Clipping: Less-Biased Variance Reduction in Off-Policy Evaluation](https://mlsanity.com/paper/2309.01120/) (2023)
- [Automated Off-Policy Estimator Selection via Supervised Learning](https://mlsanity.com/paper/2406.18022/) (2024)
- [Clustering Context in Off-Policy Evaluation](https://mlsanity.com/paper/2502.21304/) (2025)
- [Off-Policy Evaluation and Learning for Matching Markets](https://mlsanity.com/paper/2507.13608/) (2025)

## Abstract (arXiv)

We develop a generic data-driven method for estimator selection in off-policy policy evaluation settings. We establish a strong performance guarantee for the method, showing that it is competitive with the oracle estimator, up to a constant factor. Via in-depth case studies in contextual bandits and reinforcement learning, we demonstrate the generality and applicability of the method. We also perform comprehensive experiments, demonstrating the empirical efficacy of our approach and comparing with related approaches. In both case studies, our method compares favorably with existing methods.