# Counterfactual Situation Testing: From Single to Multidimensional Discrimination

arXiv: https://arxiv.org/abs/2502.01267 · HTML: https://mlsanity.com/paper/2502.01267/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 99 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Builds on (most cited first, 30 of 32)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Algorithmic Recourse: from Counterfactual Explanations to Interventions](https://mlsanity.com/paper/2002.06278/) (2020)
- [Avoiding Discrimination through Causal Reasoning](https://mlsanity.com/paper/1706.02744/) (2017)
- [Path-Specific Counterfactual Fairness](https://mlsanity.com/paper/1802.08139/) (2018)
- [Algorithmic recourse under imperfect causal knowledge: a probabilistic approach](https://mlsanity.com/paper/2006.06831/) (2020)
- [Fairness Testing: Testing Software for Discrimination](https://mlsanity.com/paper/1709.03221/) (2017)
- [Towards a Critical Race Methodology in Algorithmic Fairness](https://mlsanity.com/paper/1912.03593/) (2019)
- [Fair Regression with Wasserstein Barycenters](https://mlsanity.com/paper/2006.07286/) (2020)
- [Fairness in Credit Scoring: Assessment, Implementation and Profit Implications](https://mlsanity.com/paper/2103.01907/) (2021)
- [Machine Learning Methods Economists Should Know About](https://mlsanity.com/paper/1903.10075/) (2019)
- [Fairness Testing: A Comprehensive Survey and Analysis of Trends](https://mlsanity.com/paper/2207.10223/) (2022)
- [Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation](https://mlsanity.com/paper/2205.04610/) (2022)
- [The Use and Misuse of Counterfactuals in Ethical Machine Learning](https://mlsanity.com/paper/2102.05085/) (2021)
- [Discrimination in the Age of Algorithms](https://mlsanity.com/paper/1902.03731/) (2019)
- [FlipTest: Fairness Testing via Optimal Transport](https://mlsanity.com/paper/1906.09218/) (2019)
- [What's Sex Got To Do With Fair Machine Learning?](https://mlsanity.com/paper/2006.01770/) (2020)
- [The Sensitivity of Counterfactual Fairness to Unmeasured Confounding](https://mlsanity.com/paper/1907.01040/) (2019)
- [Fair Data Adaptation with Quantile Preservation](https://mlsanity.com/paper/1911.06685/) (2019)
- [Encoding high-cardinality string categorical variables](https://mlsanity.com/paper/1907.01860/) (2019)
- [Automated Test Generation to Detect Individual Discrimination in AI Models](https://mlsanity.com/paper/1809.03260/) (2018)
- [On Multi-Cause Causal Inference with Unobserved Confounding: Counterexamples, Impossibility, and Alternatives](https://mlsanity.com/paper/1902.10286/) (2019)
- [Multi-dimensional discrimination in Law and Machine Learning -- A comparative overview](https://mlsanity.com/paper/2302.05995/) (2023)
- [Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law is not a Decision Tree](https://mlsanity.com/paper/2305.13938/) (2023)
- [The Long Arc of Fairness: Formalisations and Ethical Discourse](https://mlsanity.com/paper/2203.06038/) (2022)
- [Causal Inference for Social Discrimination Reasoning](https://mlsanity.com/paper/1608.03735/) (2016)
- [Counterfactual Situation Testing: Uncovering Discrimination under Fairness given the Difference](https://mlsanity.com/paper/2302.11944/) (2023)
- [What is Proxy Discrimination?](https://mlsanity.com/paper/2205.05265/) (2022)

## Built on by (most cited first, 2 of 2)

- [Intersectional Divergence: Measuring Fairness in Regression](https://mlsanity.com/paper/2505.00830/) (2025)
- [Implementing Causal Perception: Competing SCMs and Situated Fairness](https://mlsanity.com/paper/2608.03917/) (2026)

## Abstract (arXiv)

We present counterfactual situation testing (CST), a causal data mining framework for detecting individual discrimination in a dataset of classifier decisions. CST answers the question ``what would have been the model outcome had the individual, or complainant, been of a different protected status?'' It extends the legally-grounded situation testing (ST) of Thanh et al. (2011) by operationalizing the notion of "fairness given the difference" via counterfactual reasoning. ST finds for each complainant similar protected and non-protected instances in the dataset; constructs, respectively, a control and test group; and compares the groups such that a difference in model outcomes implies a potential case of individual discrimination. CST, instead, avoids this idealized comparison by establishing the test group on the complainant's generated counterfactual, which reflects how the protected attribute when changed influences other seemingly neutral attributes of the complainant. Under CST we test for discrimination for each complainant by comparing similar individuals within the control and test group but dissimilar individuals across these groups. We consider single (e.g.,~gender) and multidimensional (e.g.,~gender and race) discrimination testing. For multidimensional discrimination we study multiple and intersectional discrimination and, as feared by legal scholars, find evidence that the former fails to account for the latter kind. Using a k-nearest neighbor implementation, we showcase CST on synthetic and real data. Experimental results show that CST uncovers a higher number of cases than ST, even when the model is counterfactually fair. CST, in fact, extends counterfactual fairness (CF) of Kusner et al. (2017) by equipping CF with confidence intervals, which we report for all experiments.