# Off-Policy Evaluation with Out-of-Sample Guarantees

arXiv: https://arxiv.org/abs/2301.08649 · HTML: https://mlsanity.com/paper/2301.08649/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 20 Jan 2023.

## Builds on (most cited first, 13 of 13)

- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach](https://mlsanity.com/paper/2111.12161/) (2021)
- [Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding](https://mlsanity.com/paper/2003.05623/) (2020)
- [Balanced Policy Evaluation and Learning](https://mlsanity.com/paper/1705.07384/) (2017)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap](https://mlsanity.com/paper/1711.11286/) (2017)
- [Off-Policy Risk Assessment in Contextual Bandits](https://mlsanity.com/paper/2104.08977/) (2021)
- [Conformal Off-Policy Prediction in Contextual Bandits](https://mlsanity.com/paper/2206.04405/) (2022)
- [Characterization of Overlap in Observational Studies](https://mlsanity.com/paper/1907.04138/) (2019)
- [Learning Robust Decision Policies from Observational Data](https://mlsanity.com/paper/2006.02355/) (2020)
- [Cross-screening in observational studies that test many hypotheses](https://mlsanity.com/paper/1703.02078/) (2017)

## Built on by (most cited first, 3 of 3)

- [Predictive Performance Comparison of Decision Policies Under Confounding](https://mlsanity.com/paper/2404.00848/) (2024)
- [Externally Valid Policy Evaluation Combining Trial and Observational Data](https://mlsanity.com/paper/2310.14763/) (2023)
- [Policy Learning with Observational Data: The Case of Hepatitis C Treatment for HIV/HCV Co-Infected Patients](https://mlsanity.com/paper/2605.16593/) (2026)

## Abstract (arXiv)

We consider the problem of evaluating the performance of a decision policy using past observational data. The outcome of a policy is measured in terms of a loss (aka. disutility or negative reward) and the main problem is making valid inferences about its out-of-sample loss when the past data was observed under a different and possibly unknown policy. Using a sample-splitting method, we show that it is possible to draw such inferences with finite-sample coverage guarantees about the entire loss distribution, rather than just its mean. Importantly, the method takes into account model misspecifications of the past policy - including unmeasured confounding. The evaluation method can be used to certify the performance of a policy using observational data under a specified range of credible model assumptions.