# Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting

arXiv: https://arxiv.org/abs/2006.10460 · HTML: https://mlsanity.com/paper/2006.10460/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 30 of 38)

- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters](https://mlsanity.com/paper/2205.13703/) (2022)
- [Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings](https://mlsanity.com/paper/2107.11003/) (2021)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Bootstrapping Fitted Q-Evaluation for Off-Policy Inference](https://mlsanity.com/paper/2102.03607/) (2021)
- [A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2111.06784/) (2021)
- [Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequality](https://mlsanity.com/paper/2212.09900/) (2022)
- [Online Bootstrap Inference For Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2108.03706/) (2021)
- [Conformal Off-Policy Prediction in Contextual Bandits](https://mlsanity.com/paper/2206.04405/) (2022)
- [Policy-Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2211.13904/) (2022)
- [PAC-Bayesian Offline Contextual Bandits With Guarantees](https://mlsanity.com/paper/2210.13132/) (2022)
- [Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning](https://mlsanity.com/paper/2405.14335/) (2024)
- [Exponential Smoothing for Off-Policy Learning](https://mlsanity.com/paper/2305.15877/) (2023)
- [Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings](https://mlsanity.com/paper/2105.06029/) (2021)
- [BR-SNIS: Bias Reduced Self-Normalized Importance Sampling](https://mlsanity.com/paper/2207.06364/) (2022)
- [Uncertainty Quantification for Fairness in Two-Stage Recommender Systems](https://mlsanity.com/paper/2205.15436/) (2022)
- [Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation](https://mlsanity.com/paper/2405.21043/) (2024)
- [High-Confidence Off-Policy (or Counterfactual) Variance Estimation](https://mlsanity.com/paper/2101.09847/) (2021)
- [PAC-Bayes Bounds for Bandit Problems: A Survey and Experimental Comparison](https://mlsanity.com/paper/2211.16110/) (2022)
- [Conformal Off-Policy Evaluation in Markov Decision Processes](https://mlsanity.com/paper/2304.02574/) (2023)
- [Bayesian Off-Policy Evaluation and Learning for Large Action Spaces](https://mlsanity.com/paper/2402.14664/) (2024)
- [Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with It](https://mlsanity.com/paper/2404.15084/) (2024)
- [Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approach](https://mlsanity.com/paper/2505.20130/) (2025)
- [Improved Offline Contextual Bandits with Second-Order Bounds: Betting and Freezing](https://mlsanity.com/paper/2502.10826/) (2025)
- [Pessimistic Model Selection for Offline Deep Reinforcement Learning](https://mlsanity.com/paper/2111.14346/) (2021)
- [Counterfactual Learning with General Data-generating Policies](https://mlsanity.com/paper/2212.01925/) (2022)
- [Probabilistic Offline Policy Ranking with Approximate Bayesian Computation](https://mlsanity.com/paper/2312.11551/) (2023)
- [Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling](https://mlsanity.com/paper/2406.03434/) (2024)
- [On the Reuse Bias in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2209.07074/) (2022)
- [Semi-supervised Batch Learning From Logged Data](https://mlsanity.com/paper/2209.07148/) (2022)

## Abstract (arXiv)

We consider off-policy evaluation in the contextual bandit setting for the purpose of obtaining a robust off-policy selection strategy, where the selection strategy is evaluated based on the value of the chosen policy in a set of proposal (target) policies. We propose a new method to compute a lower bound on the value of an arbitrary target policy given some logged data in contextual bandits for a desired coverage. The lower bound is built around the so-called Self-normalized Importance Weighting (SN) estimator. It combines the use of a semi-empirical Efron-Stein tail inequality to control the concentration and a new multiplicative (rather than additive) control of the bias. The new approach is evaluated on a number of synthetic and real datasets and is found to be superior to its main competitors, both in terms of tightness of the confidence intervals and the quality of the policies chosen.