# Counterfactual Situation Testing: Uncovering Discrimination under Fairness given the Difference

arXiv: https://arxiv.org/abs/2302.11944 · HTML: https://mlsanity.com/paper/2302.11944/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 79 references that are in the map.
- First posted on arXiv 23 Feb 2023.

## Builds on (most cited first, 22 of 22)

- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Algorithmic Recourse: from Counterfactual Explanations to Interventions](https://mlsanity.com/paper/2002.06278/) (2020)
- [Avoiding Discrimination through Causal Reasoning](https://mlsanity.com/paper/1706.02744/) (2017)
- [Path-Specific Counterfactual Fairness](https://mlsanity.com/paper/1802.08139/) (2018)
- [Fairness Testing: Testing Software for Discrimination](https://mlsanity.com/paper/1709.03221/) (2017)
- [Towards a Critical Race Methodology in Algorithmic Fairness](https://mlsanity.com/paper/1912.03593/) (2019)
- [Fair Regression with Wasserstein Barycenters](https://mlsanity.com/paper/2006.07286/) (2020)
- [Machine Learning Methods Economists Should Know About](https://mlsanity.com/paper/1903.10075/) (2019)
- [Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation](https://mlsanity.com/paper/2205.04610/) (2022)
- [Survey on Causal-based Machine Learning Fairness Notions](https://mlsanity.com/paper/2010.09553/) (2020)
- [The Use and Misuse of Counterfactuals in Ethical Machine Learning](https://mlsanity.com/paper/2102.05085/) (2021)
- [Discrimination in the Age of Algorithms](https://mlsanity.com/paper/1902.03731/) (2019)
- [FlipTest: Fairness Testing via Optimal Transport](https://mlsanity.com/paper/1906.09218/) (2019)
- [Causal Fairness Analysis](https://mlsanity.com/paper/2207.11385/) (2022)
- [What's Sex Got To Do With Fair Machine Learning?](https://mlsanity.com/paper/2006.01770/) (2020)
- [The Sensitivity of Counterfactual Fairness to Unmeasured Confounding](https://mlsanity.com/paper/1907.01040/) (2019)
- [Automated Test Generation to Detect Individual Discrimination in AI Models](https://mlsanity.com/paper/1809.03260/) (2018)
- [On Multi-Cause Causal Inference with Unobserved Confounding: Counterexamples, Impossibility, and Alternatives](https://mlsanity.com/paper/1902.10286/) (2019)
- [Causal Inference for Social Discrimination Reasoning](https://mlsanity.com/paper/1608.03735/) (2016)
- [What is Proxy Discrimination?](https://mlsanity.com/paper/2205.05265/) (2022)

## Built on by (most cited first, 8 of 8)

- [Fairness Evaluation with Item Response Theory](https://mlsanity.com/paper/2411.02414/) (2024)
- [Causal Fair Machine Learning via Rank-Preserving Interventional Distributions](https://mlsanity.com/paper/2307.12797/) (2023)
- [Toward A Causal Framework for Modeling Perception](https://mlsanity.com/paper/2401.13408/) (2024)
- [Mutatis Mutandis: Revisiting the Comparator in Discrimination Testing](https://mlsanity.com/paper/2405.13693/) (2024)
- [Formalising Anti-Discrimination Law in Automated Decision Systems](https://mlsanity.com/paper/2407.00400/) (2024)
- [Counterfactual Situation Testing: From Single to Multidimensional Discrimination](https://mlsanity.com/paper/2502.01267/) (2025)
- [Privilege Scores](https://mlsanity.com/paper/2502.01211/) (2025)
- [What's the Problem, Linda? The Conjunction Fallacy as a Fairness Problem](https://mlsanity.com/paper/2305.09535/) (2023)

## Abstract (arXiv)

We present counterfactual situation testing (CST), a causal data mining framework for detecting discrimination in classifiers. CST aims to answer in an actionable and meaningful way the intuitive question "what would have been the model outcome had the individual, or complainant, been of a different protected status?" It extends the legally-grounded situation testing of Thanh et al. (2011) by operationalizing the notion of fairness given the difference using counterfactual reasoning. For any complainant, we find and compare similar protected and non-protected instances in the dataset used by the classifier to construct a control and test group, where a difference between the decision outcomes of the two groups implies potential individual discrimination. Unlike situation testing, which builds both groups around the complainant, we build the test group on the complainant's counterfactual generated using causal knowledge. The counterfactual is intended to reflect how the protected attribute when changed affects the seemingly neutral attributes used by the classifier, which is taken for granted in many frameworks for discrimination. Under CST, we compare similar individuals within each group but dissimilar individuals across both groups due to the possible difference between the complainant and its counterfactual. Evaluating our framework on two classification scenarios, we show that it uncovers a greater number of cases than situation testing, even when the classifier satisfies the counterfactual fairness condition of Kusner et al. (2017).