mlsanityData to 7 Oct 2026

Robust Design and Evaluation of Predictive Algorithms under Unobserved Confounding

arXiv 2212.09844 ↗First posted 19 Dec 2022econ.EM cs.CY cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Predictive algorithms inform consequential decisions in settings with selective labels: outcomes are observed only for units selected by past decision makers. This creates an identification problem under unobserved confounding -- when selected and unselected units differ in unobserved ways that affect outcomes. We propose a framework for robust design and evaluation of predictive algorithms that bounds how much outcomes may differ between selected and unselected units with the same observed characteristics. These bounds formalize common empirical strategies including proxy outcomes and instrumental variables. Our estimators work across bounding strategies and performance measures such as conditional likelihoods, mean square error, and true/false positive rates. Using administrative data from a large Australian financial institution, we show that varying confounding assumptions substantially affects credit risk predictions and fairness evaluations across income groups.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 24 of its 76 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

24 of 76 references.

References · 24

  1. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  2. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
  3. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 328
  4. 2019Fair Regression: Quantitative Definitions and Reduction-based Algorithmscited by 168
  5. 2019Orthogonal Statistical Learningcited by 150
  6. 2021Characterizing Fairness Over the Set of Good Models Under Selective Labelscited by 65
  7. 2021Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approachcited by 62
  8. 2019Counterfactual Risk Assessments, Evaluation, and Fairnesscited by 59
  9. 2018Interval Estimation of Individual-Level Causal Effects Under Unobserved Confoundingcited by 51
  10. 2020Fairness in Risk Assessment Instruments: Post-Processing to Achieve Counterfactual Equalized Oddscited by 40
  11. 2018Bounds on the conditional and average treatment effect with unobserved confounding factorscited by 39
  12. 2017Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrapcited by 33
12 more
  1. 2021Sharp Sensitivity Analysis for Inverse Propensity Weighting via Quantile Balancingcited by 32
  2. 2021Evaluating Treatment Prioritization Rules via Rank-Weighted Average Treatment Effectscited by 29
  3. 2021Doubly-Valid/Doubly-Sharp Sensitivity Analysis for Causal Inference with Unmeasured Confoundingcited by 28
  4. 2021Long Story Short: Omitted Variable Bias in Causal Machine Learningcited by 23
  5. 2018Flexible sensitivity analysis for observational studies without observable implicationscited by 18
  6. 2021FADE: FAir Double Ensemble Learning for Observable and Counterfactual Outcomescited by 13
  7. 2022Algorithmic Fairness and Vertical Equity: Income Fairness with IRS Tax Audit Modelscited by 12
  8. 2021Semiparametric sensitivity analysis: unmeasured confounding in observational studiescited by 8
  9. 2017Inference on Breakdown Frontierscited by 4
  10. 2017Identification of Treatment Effects under Conditional Partial Independencecited by 3
  11. 2022Optimal Decision Rules when Payoffs are Partially Identifiedcited by 3
  12. 2022Inference with Imputed Data: The Allure of Making Stuff Upcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 1; 2026: 2.

7 papers.

  1. 2023Counterfactual Prediction Under Outcome Measurement Errorcited by 10
  2. 2024What Constitutes a Less Discriminatory Algorithm?cited by 8
  3. 2025Efficient and Sharp Off-Policy Learning under Unobserved Confoundingcited by 5
  4. 2024Predictive Performance Comparison of Decision Policies Under Confoundingcited by 2
  5. 2023Bridging the gap: Towards an Expanded Toolkit for AI-driven Decision-Making in the Public Sectorcited by 0
  6. 2026Falsifying Discriminant Validity of Predictive Algorithmscited by 0
  7. 2026The Exceedance Design Effect: Effective Sample Size for Thresholds under Clusteringcited by 0