# Bias In, Bias Out? Evaluating the Folk Wisdom

arXiv: https://arxiv.org/abs/1909.08518 · HTML: https://mlsanity.com/paper/1909.08518/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2019.

## Built on by (most cited first, 7 of 7)

- [Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making](https://mlsanity.com/paper/2302.06503/) (2023)
- [Bias Propagation in Federated Learning](https://mlsanity.com/paper/2309.02160/) (2023)
- [Equalizing Credit Opportunity in Algorithms: Aligning Algorithmic Fairness Research with U.S. Fair Lending Regulation](https://mlsanity.com/paper/2210.02516/) (2022)
- [From Biased Selective Labels to Pseudo-Labels: An Expectation-Maximization Framework for Learning from Biased Decisions](https://mlsanity.com/paper/2406.18865/) (2024)
- [Unpacking the Black Box: Regulating Algorithmic Decisions](https://mlsanity.com/paper/2110.03443/) (2021)
- [On Optimal Steering to Achieve Exact Fairness](https://mlsanity.com/paper/2509.15759/) (2025)
- [Provable Detection of Propagating Sampling Bias in Prediction Models](https://mlsanity.com/paper/2302.06752/) (2023)

## Abstract (arXiv)

We evaluate the folk wisdom that algorithmic decision rules trained on data produced by biased human decision-makers necessarily reflect this bias. We consider a setting where training labels are only generated if a biased decision-maker takes a particular action, and so "biased" training data arise due to discriminatory selection into the training data. In our baseline model, the more biased the decision-maker is against a group, the more the algorithmic decision rule favors that group. We refer to this phenomenon as "bias reversal." We then clarify the conditions that give rise to bias reversal. Whether a prediction algorithm reverses or inherits bias depends critically on how the decision-maker affects the training data as well as the label used in training. We illustrate our main theoretical results in a simulation study applied to the New York City Stop, Question and Frisk dataset.