# Auditing Fairness under Unobserved Confounding

arXiv: https://arxiv.org/abs/2403.14713 · HTML: https://mlsanity.com/paper/2403.14713/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 36 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 11 of 11)

- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Semiparametric doubly robust targeted double machine learning: a review](https://mlsanity.com/paper/2203.06469/) (2022)
- [Counterfactual Risk Assessments, Evaluation, and Fairness](https://mlsanity.com/paper/1909.00066/) (2019)
- [Causal Fairness Analysis](https://mlsanity.com/paper/2207.11385/) (2022)
- [Semiparametric counterfactual density estimation](https://mlsanity.com/paper/2102.12034/) (2021)
- [Fairness in Risk Assessment Instruments: Post-Processing to Achieve Counterfactual Equalized Odds](https://mlsanity.com/paper/2009.02841/) (2020)
- [Bounds on the conditional and average treatment effect with unobserved confounding factors](https://mlsanity.com/paper/1808.09521/) (2018)
- [Causal Conceptions of Fairness and their Consequences](https://mlsanity.com/paper/2207.05302/) (2022)
- [Sharp instruments for classifying compliers and generalizing causal effects](https://mlsanity.com/paper/1801.03635/) (2018)
- [Covariate-assisted bounds on causal effects with instrumental variables](https://mlsanity.com/paper/2301.12106/) (2023)

## Built on by (most cited first, 6 of 6)

- [The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning](https://mlsanity.com/paper/2410.09600/) (2024)
- [Decision-Focused Evaluation of Worst-Case Distribution Shift](https://mlsanity.com/paper/2407.03557/) (2024)
- [From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms](https://mlsanity.com/paper/2502.08166/) (2025)
- [Causality Is Key to Understand and Balance Multiple Goals in Trustworthy ML and Foundation Models](https://mlsanity.com/paper/2502.21123/) (2025)
- [Accounting for Missing Covariates in Heterogeneous Treatment Estimation](https://mlsanity.com/paper/2410.15655/) (2024)
- [Cost Efficient Fairness Audit Under Partial Feedback](https://mlsanity.com/paper/2510.03734/) (2025)

## Abstract (arXiv)

Many definitions of fairness or inequity involve unobservable causal quantities that cannot be directly estimated without strong assumptions. For instance, it is particularly difficult to estimate notions of fairness that rely on hard-to-measure concepts such as risk (e.g., quantifying whether patients at the same risk level have equal probability of treatment, regardless of group membership). Such measurements of risk can be accurately obtained when no unobserved confounders have jointly influenced past decisions and outcomes. However, in the real world, this assumption rarely holds. In this paper, we show that, surprisingly, one can still compute meaningful bounds on treatment rates for high-risk individuals (i.e., conditional on their true, textitunobserved negative outcome), even when entirely eliminating or relaxing the assumption that we observe all relevant risk factors used by decision makers. We use the fact that in many real-world settings (e.g., the release of a new treatment) we have data from prior to any allocation to derive unbiased estimates of risk. This result enables us to audit unfair outcomes of existing decision-making systems in a principled manner. We demonstrate the effectiveness of our framework with a real-world study of Paxlovid allocation, provably identifying that observed racial inequity cannot be explained by unobserved confounders of the same strength as important observed covariates.