# Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics

arXiv: https://arxiv.org/abs/2001.00089 · HTML: https://mlsanity.com/paper/2001.00089/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2019.

## Built on by (most cited first, 4 of 4)

- [Fairness Perceptions of Algorithmic Decision-Making: A Systematic Review of the Empirical Literature](https://mlsanity.com/paper/2103.12016/) (2021)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [Responding to Generative AI Technologies with Research-through-Design: The Ryelands AI Lab as an Exploratory Study](https://mlsanity.com/paper/2405.04677/) (2024)
- [The Perspective of Software Professionals on Algorithmic Racism](https://mlsanity.com/paper/2306.15133/) (2023)

## Abstract (arXiv)

Bias in machine learning has manifested injustice in several areas, such as medicine, hiring, and criminal justice. In response, computer scientists have developed myriad definitions of fairness to correct this bias in fielded algorithms. While some definitions are based on established legal and ethical norms, others are largely mathematical. It is unclear whether the general public agrees with these fairness definitions, and perhaps more importantly, whether they understand these definitions. We take initial steps toward bridging this gap between ML researchers and the public, by addressing the question: does a lay audience understand a basic definition of ML fairness? We develop a metric to measure comprehension of three such definitions--demographic parity, equal opportunity, and equalized odds. We evaluate this metric using an online survey, and investigate the relationship between comprehension and sentiment, demographics, and the definition itself.