# Prisoners of Their Own Devices: How Models Induce Data Bias in Performative Prediction

arXiv: https://arxiv.org/abs/2206.13183 · HTML: https://mlsanity.com/paper/2206.13183/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 27 Jun 2022.

## Builds on (most cited first, 14 of 14)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Fairness in Machine Learning](https://mlsanity.com/paper/2012.15816/) (2020)
- [Identifying and Correcting Label Bias in Machine Learning](https://mlsanity.com/paper/1901.04966/) (2019)
- [Fair Classification with Group-Dependent Label Noise](https://mlsanity.com/paper/2011.00379/) (2020)
- [Noise-tolerant fair classification](https://mlsanity.com/paper/1901.10837/) (2019)
- [Fairness Evaluation in Presence of Biased Noisy Labels](https://mlsanity.com/paper/2003.13808/) (2020)
- [Fair When Trained, Unfair When Deployed: Observable Fairness Measures are Unstable in Performative Prediction Settings](https://mlsanity.com/paper/2202.05049/) (2022)
- [A Sandbox Tool to Bias(Stress)-Test Fairness Algorithms](https://mlsanity.com/paper/2204.10233/) (2022)
- [Algorithmic Factors Influencing Bias in Machine Learning](https://mlsanity.com/paper/2104.14014/) (2021)
- [Unfairness Despite Awareness: Group-Fair Classification with Strategic Agents](https://mlsanity.com/paper/2112.02746/) (2021)

## Built on by (most cited first, 5 of 5)

- [Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluation](https://mlsanity.com/paper/2211.13358/) (2022)
- [Fairness-Aware Data Valuation for Supervised Learning](https://mlsanity.com/paper/2303.16963/) (2023)
- [Evaluating and Correcting Performative Effects of Decision Support Systems via Causal Domain Shift](https://mlsanity.com/paper/2403.00886/) (2024)
- [When Predictions Shape Reality: A Socio-Technical Synthesis of Performative Predictions in Machine Learning](https://mlsanity.com/paper/2601.04447/) (2026)
- [Performative Prediction with Selective Labels](https://mlsanity.com/paper/2610.08272/) (2026)

## Abstract (arXiv)

The unparalleled ability of machine learning algorithms to learn patterns from data also enables them to incorporate biases embedded within. A biased model can then make decisions that disproportionately harm certain groups in society. Much work has been devoted to measuring unfairness in static ML environments, but not in dynamic, performative prediction ones, in which most real-world use cases operate. In the latter, the predictive model itself plays a pivotal role in shaping the distribution of the data. However, little attention has been heeded to relating unfairness to these interactions. Thus, to further the understanding of unfairness in these settings, we propose a taxonomy to characterize bias in the data, and study cases where it is shaped by model behaviour. Using a real-world account opening fraud detection case study as an example, we study the dangers to both performance and fairness of two typical biases in performative prediction: distribution shifts, and the problem of selective labels.