# Robust Design and Evaluation of Predictive Algorithms under Unobserved Confounding

arXiv: https://arxiv.org/abs/2212.09844 · HTML: https://mlsanity.com/paper/2212.09844/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 76 references that are in the map.
- First posted on arXiv 19 Dec 2022.

## Builds on (most cited first, 24 of 24)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Fair Regression: Quantitative Definitions and Reduction-based Algorithms](https://mlsanity.com/paper/1905.12843/) (2019)
- [Orthogonal Statistical Learning](https://mlsanity.com/paper/1901.09036/) (2019)
- [Characterizing Fairness Over the Set of Good Models Under Selective Labels](https://mlsanity.com/paper/2101.00352/) (2021)
- [Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach](https://mlsanity.com/paper/2111.12161/) (2021)
- [Counterfactual Risk Assessments, Evaluation, and Fairness](https://mlsanity.com/paper/1909.00066/) (2019)
- [Interval Estimation of Individual-Level Causal Effects Under Unobserved Confounding](https://mlsanity.com/paper/1810.02894/) (2018)
- [Fairness in Risk Assessment Instruments: Post-Processing to Achieve Counterfactual Equalized Odds](https://mlsanity.com/paper/2009.02841/) (2020)
- [Bounds on the conditional and average treatment effect with unobserved confounding factors](https://mlsanity.com/paper/1808.09521/) (2018)
- [Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap](https://mlsanity.com/paper/1711.11286/) (2017)
- [Sharp Sensitivity Analysis for Inverse Propensity Weighting via Quantile Balancing](https://mlsanity.com/paper/2102.04543/) (2021)
- [Evaluating Treatment Prioritization Rules via Rank-Weighted Average Treatment Effects](https://mlsanity.com/paper/2111.07966/) (2021)
- [Doubly-Valid/Doubly-Sharp Sensitivity Analysis for Causal Inference with Unmeasured Confounding](https://mlsanity.com/paper/2112.11449/) (2021)
- [Long Story Short: Omitted Variable Bias in Causal Machine Learning](https://mlsanity.com/paper/2112.13398/) (2021)
- [Flexible sensitivity analysis for observational studies without observable implications](https://mlsanity.com/paper/1809.00399/) (2018)
- [FADE: FAir Double Ensemble Learning for Observable and Counterfactual Outcomes](https://mlsanity.com/paper/2109.00173/) (2021)
- [Algorithmic Fairness and Vertical Equity: Income Fairness with IRS Tax Audit Models](https://mlsanity.com/paper/2206.09875/) (2022)
- [Semiparametric sensitivity analysis: unmeasured confounding in observational studies](https://mlsanity.com/paper/2104.08300/) (2021)
- [Inference on Breakdown Frontiers](https://mlsanity.com/paper/1705.04765/) (2017)
- [Identification of Treatment Effects under Conditional Partial Independence](https://mlsanity.com/paper/1707.09563/) (2017)
- [Optimal Decision Rules when Payoffs are Partially Identified](https://mlsanity.com/paper/2204.11748/) (2022)
- [Inference with Imputed Data: The Allure of Making Stuff Up](https://mlsanity.com/paper/2205.07388/) (2022)

## Built on by (most cited first, 8 of 8)

- [Counterfactual Prediction Under Outcome Measurement Error](https://mlsanity.com/paper/2302.11121/) (2023)
- [What Constitutes a Less Discriminatory Algorithm?](https://mlsanity.com/paper/2412.18138/) (2024)
- [Efficient and Sharp Off-Policy Learning under Unobserved Confounding](https://mlsanity.com/paper/2502.13022/) (2025)
- [Predictive Performance Comparison of Decision Policies Under Confounding](https://mlsanity.com/paper/2404.00848/) (2024)
- [Bridging the gap: Towards an Expanded Toolkit for AI-driven Decision-Making in the Public Sector](https://mlsanity.com/paper/2310.19091/) (2023)
- [Falsifying Discriminant Validity of Predictive Algorithms](https://mlsanity.com/paper/2601.17146/) (2026)
- [Identification and Inference for Algorithmic Frontiers with Selective Labels](https://mlsanity.com/paper/2606.14977/) (2026)
- [The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering](https://mlsanity.com/paper/2608.21262/) (2026)

## Abstract (arXiv)

Predictive algorithms inform consequential decisions in settings with selective labels: outcomes are observed only for units selected by past decision makers. This creates an identification problem under unobserved confounding -- when selected and unselected units differ in unobserved ways that affect outcomes. We propose a framework for robust design and evaluation of predictive algorithms that bounds how much outcomes may differ between selected and unselected units with the same observed characteristics. These bounds formalize common empirical strategies including proxy outcomes and instrumental variables. Our estimators work across bounding strategies and performance measures such as conditional likelihoods, mean square error, and true/false positive rates. Using administrative data from a large Australian financial institution, we show that varying confounding assumptions substantially affects credit risk predictions and fairness evaluations across income groups.