# Externally Valid Policy Evaluation Combining Trial and Observational Data

arXiv: https://arxiv.org/abs/2310.14763 · HTML: https://mlsanity.com/paper/2310.14763/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 57 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Calibration tests in multi-class classification: A unifying framework](https://mlsanity.com/paper/1910.11385/) (2019)
- [External Validity: From Do-Calculus to Transportability Across Populations](https://mlsanity.com/paper/1503.01603/) (2015)
- [Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach](https://mlsanity.com/paper/2111.12161/) (2021)
- [Causal inference methods for combining randomized trials and observational studies: a review](https://mlsanity.com/paper/2011.08047/) (2020)
- [A Review of Generalizability and Transportability](https://mlsanity.com/paper/2102.11904/) (2021)
- [Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap](https://mlsanity.com/paper/1711.11286/) (2017)
- [Sharp Sensitivity Analysis for Inverse Propensity Weighting via Quantile Balancing](https://mlsanity.com/paper/2102.04543/) (2021)
- [Off-Policy Risk Assessment in Contextual Bandits](https://mlsanity.com/paper/2104.08977/) (2021)
- [Sense and Sensitivity Analysis: Simple Post-Hoc Analysis of Bias Due to Unobserved Confounding](https://mlsanity.com/paper/2003.01747/) (2020)
- [Flexible sensitivity analysis for observational studies without observable implications](https://mlsanity.com/paper/1809.00399/) (2018)
- [Covariate Balancing Sensitivity Analysis for Extrapolating Randomized Trials across Locations](https://mlsanity.com/paper/2112.04723/) (2021)
- [Sensitivity Analysis in the Generalization of Experimental Results](https://mlsanity.com/paper/2202.03408/) (2022)
- [Generalizing trial evidence to target populations in non-nested designs: Applications to AIDS clinical trials](https://mlsanity.com/paper/2103.04907/) (2021)
- [Off-Policy Evaluation with Out-of-Sample Guarantees](https://mlsanity.com/paper/2301.08649/) (2023)

## Abstract (arXiv)

Randomized trials are widely considered as the gold standard for evaluating the effects of decision policies. Trial data is, however, drawn from a population which may differ from the intended target population and this raises a problem of external validity (aka. generalizability). In this paper we seek to use trial data to draw valid inferences about the outcome of a policy on the target population. Additional covariate data from the target population is used to model the sampling of individuals in the trial study. We develop a method that yields certifiably valid trial-based policy evaluations under any specified range of model miscalibrations. The method is nonparametric and the validity is assured even with finite samples. The certified policy evaluations are illustrated using both simulated and real data.