# Predictive Performance Comparison of Decision Policies Under Confounding

arXiv: https://arxiv.org/abs/2404.00848 · HTML: https://mlsanity.com/paper/2404.00848/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 74 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 30 of 35)

- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Are Anchor Points Really Indispensable in Label-Noise Learning?](https://mlsanity.com/paper/1906.00189/) (2019)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [An Introduction to Proximal Causal Learning](https://mlsanity.com/paper/2009.10982/) (2020)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination](https://mlsanity.com/paper/1906.00285/) (2019)
- [Confounding-Robust Policy Improvement](https://mlsanity.com/paper/1805.08593/) (2018)
- [Classification with Asymmetric Label Noise: Consistency and Maximal Denoising](https://mlsanity.com/paper/1303.1208/) (2013)
- [Characterizing Fairness Over the Set of Good Models Under Selective Labels](https://mlsanity.com/paper/2101.00352/) (2021)
- [Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding](https://mlsanity.com/paper/2003.05623/) (2020)
- [Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning](https://mlsanity.com/paper/2002.04518/) (2020)
- [Semiparametric counterfactual density estimation](https://mlsanity.com/paper/2102.12034/) (2021)
- [Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluation](https://mlsanity.com/paper/2106.03907/) (2021)
- [Fairness Evaluation in Presence of Biased Noisy Labels](https://mlsanity.com/paper/2003.13808/) (2020)
- [Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting](https://mlsanity.com/paper/2006.10460/) (2020)
- [A confounding bridge approach for double negative control inference on causal effects](https://mlsanity.com/paper/1808.04945/) (2018)
- [Offline Policy Selection under Uncertainty](https://mlsanity.com/paper/2012.06919/) (2020)
- [A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms](https://mlsanity.com/paper/2206.14983/) (2022)
- [Active Offline Policy Selection](https://mlsanity.com/paper/2106.10251/) (2021)
- [Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making](https://mlsanity.com/paper/2302.06503/) (2023)
- [Sharp instruments for classifying compliers and generalizing causal effects](https://mlsanity.com/paper/1801.03635/) (2018)
- [Bandits with Partially Observable Confounded Data](https://mlsanity.com/paper/2006.06731/) (2020)
- [Learning under selective labels in the presence of expert consistency](https://mlsanity.com/paper/1807.00905/) (2018)
- [Estimating Optimal Treatment Rules with an Instrumental Variable: A Partial Identification Learning Approach](https://mlsanity.com/paper/2002.02579/) (2020)
- [Generalizing Off-Policy Learning under Sample Selection Bias](https://mlsanity.com/paper/2112.01387/) (2021)
- [Robust Design and Evaluation of Predictive Algorithms under Unobserved Confounding](https://mlsanity.com/paper/2212.09844/) (2022)
- [Domain constraints improve risk prediction when outcome data is missing](https://mlsanity.com/paper/2312.03878/) (2023)
- [Partial Identification of Causal Effects Using Proxy Variables](https://mlsanity.com/paper/2304.04374/) (2023)
- [Selecting and ranking individualized treatment rules with unmeasured confounding](https://mlsanity.com/paper/2002.10436/) (2020)

## Built on by (most cited first, 2 of 2)

- [Efficient and Sharp Off-Policy Learning under Unobserved Confounding](https://mlsanity.com/paper/2502.13022/) (2025)
- [Nous: Learning and Certifying Memory Decisions Before Source Calibration](https://mlsanity.com/paper/2610.00094/) (2026)

## Abstract (arXiv)

Predictive models are often introduced to decision-making tasks under the rationale that they improve performance over an existing decision-making policy. However, it is challenging to compare predictive performance against an existing decision-making policy that is generally under-specified and dependent on unobservable factors. These sources of uncertainty are often addressed in practice by making strong assumptions about the data-generating mechanism. In this work, we propose a method to compare the predictive performance of decision policies under a variety of modern identification approaches from the causal inference and off-policy evaluation literatures (e.g., instrumental variable, marginal sensitivity model, proximal variable). Key to our method is the insight that there are regions of uncertainty that we can safely ignore in the policy comparison. We develop a practical approach for finite-sample estimation of regret intervals under no assumptions on the parametric form of the status quo policy. We verify our framework theoretically and via synthetic data experiments. We conclude with a real-world application using our framework to support a pre-deployment evaluation of a proposed modification to a healthcare enrollment policy.