# Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AI

arXiv: https://arxiv.org/abs/2005.05906 · HTML: https://mlsanity.com/paper/2005.05906/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 30 of 81)

- [Large Language Models in Law: A Survey](https://mlsanity.com/paper/2312.03718/) (2023)
- [Regulating ChatGPT and other Large Generative AI Models](https://mlsanity.com/paper/2302.02337/) (2023)
- [Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey](https://mlsanity.com/paper/2309.13933/) (2023)
- [LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions](https://mlsanity.com/paper/2406.08824/) (2024)
- [Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud](https://mlsanity.com/paper/2109.03285/) (2021)
- [D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias](https://mlsanity.com/paper/2208.05126/) (2022)
- [Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML](https://mlsanity.com/paper/2303.08485/) (2023)
- [Out of Context: Investigating the Bias and Fairness Concerns of "Artificial Intelligence as a Service"](https://mlsanity.com/paper/2302.01448/) (2023)
- [Multi-dimensional discrimination in Law and Machine Learning -- A comparative overview](https://mlsanity.com/paper/2302.05995/) (2023)
- [Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law is not a Decision Tree](https://mlsanity.com/paper/2305.13938/) (2023)
- [System Cards for AI-Based Decision-Making for Public Policy](https://mlsanity.com/paper/2203.04754/) (2022)
- [Algorithmic Arbitrariness in Content Moderation](https://mlsanity.com/paper/2402.16979/) (2024)
- [Towards Involving End-users in Interactive Human-in-the-loop AI Fairness](https://mlsanity.com/paper/2204.10464/) (2022)
- [The European AI Liability Directives -- Critique of a Half-Hearted Approach and Lessons for the Future](https://mlsanity.com/paper/2211.13960/) (2022)
- [Evaluating the Fairness of Discriminative Foundation Models in Computer Vision](https://mlsanity.com/paper/2310.11867/) (2023)
- [OxonFair: A Flexible Toolkit for Algorithmic Fairness](https://mlsanity.com/paper/2407.13710/) (2024)
- [Establishing and Evaluating Trustworthy AI: Overview and Research Challenges](https://mlsanity.com/paper/2411.09973/) (2024)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [How to be fair? A study of label and selection bias](https://mlsanity.com/paper/2403.14282/) (2024)
- [Are Two Heads the Same as One? Identifying Disparate Treatment in Fair Neural Networks](https://mlsanity.com/paper/2204.04440/) (2022)
- [BeFair: Addressing Fairness in the Banking Sector](https://mlsanity.com/paper/2102.02137/) (2021)
- [fairret: a Framework for Differentiable Fairness Regularization Terms](https://mlsanity.com/paper/2310.17256/) (2023)
- [Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions](https://mlsanity.com/paper/2107.09546/) (2021)
- [Disentangling and Operationalizing AI Fairness at LinkedIn](https://mlsanity.com/paper/2306.00025/) (2023)
- [Fairness-Aware Meta-Learning via Nash Bargaining](https://mlsanity.com/paper/2406.07029/) (2024)
- [Compatibility of Fairness Metrics with EU Non-Discrimination Laws: Demographic Parity & Conditional Demographic Disparity](https://mlsanity.com/paper/2306.08394/) (2023)
- [Implications of the AI Act for Non-Discrimination Law and Algorithmic Fairness](https://mlsanity.com/paper/2403.20089/) (2024)
- [Unlawful Proxy Discrimination: A Framework for Challenging Inherently Discriminatory Algorithms](https://mlsanity.com/paper/2404.14050/) (2024)
- [Regulating Gatekeeper AI and Data: Transparency, Access, and Fairness under the DMA, the GDPR, and beyond](https://mlsanity.com/paper/2212.04997/) (2022)
- [Bias and Fairness in Computer Vision Applications of the Criminal Justice System](https://mlsanity.com/paper/2208.03209/) (2022)

## Abstract (arXiv)

This article identifies a critical incompatibility between European notions of discrimination and existing statistical measures of fairness. First, we review the evidential requirements to bring a claim under EU non-discrimination law. Due to the disparate nature of algorithmic and human discrimination, the EU's current requirements are too contextual, reliant on intuition, and open to judicial interpretation to be automated. Second, we show how the legal protection offered by non-discrimination law is challenged when AI, not humans, discriminate. Humans discriminate due to negative attitudes (e.g. stereotypes, prejudice) and unintentional biases (e.g. organisational practices or internalised stereotypes) which can act as a signal to victims that discrimination has occurred. Finally, we examine how existing work on fairness in machine learning lines up with procedures for assessing cases under EU non-discrimination law. We propose "conditional demographic disparity" (CDD) as a standard baseline statistical measurement that aligns with the European Court of Justice's "gold standard." Establishing a standard set of statistical evidence for automated discrimination cases can help ensure consistent procedures for assessment, but not judicial interpretation, of cases involving AI and automated systems. Through this proposal for procedural regularity in the identification and assessment of automated discrimination, we clarify how to build considerations of fairness into automated systems as far as possible while still respecting and enabling the contextual approach to judicial interpretation practiced under EU non-discrimination law. N.B. Abridged abstract