# Runaway Feedback Loops in Predictive Policing

arXiv: https://arxiv.org/abs/1706.09847 · HTML: https://mlsanity.com/paper/1706.09847/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2017.

## Built on by (most cited first, 30 of 97)

- [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022)
- [Data Feedback Loops: Model-driven Amplification of Dataset Biases](https://mlsanity.com/paper/2209.03942/) (2022)
- [Socially Responsible AI Algorithms: Issues, Purposes, and Challenges](https://mlsanity.com/paper/2101.02032/) (2021)
- [Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias](https://mlsanity.com/paper/2403.07857/) (2024)
- [Performative Prediction: Past and Future](https://mlsanity.com/paper/2310.16608/) (2023)
- [Multiplayer Performative Prediction: Learning in Decision-Dependent Games](https://mlsanity.com/paper/2201.03398/) (2022)
- [Performative Prediction with Neural Networks](https://mlsanity.com/paper/2304.06879/) (2023)
- [Position: Model Collapse Does Not Mean What You Think](https://mlsanity.com/paper/2503.03150/) (2025)
- [Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans](https://mlsanity.com/paper/2401.08495/) (2024)
- [A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms](https://mlsanity.com/paper/2206.14983/) (2022)
- [Long-Term Fairness with Unknown Dynamics](https://mlsanity.com/paper/2304.09362/) (2023)
- [Human-AI Coevolution](https://mlsanity.com/paper/2306.13723/) (2023)
- [Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making](https://mlsanity.com/paper/2302.06503/) (2023)
- [Counterfactual Explanations for Machine Learning: Challenges Revisited](https://mlsanity.com/paper/2106.07756/) (2021)
- [Unintended Selection: Persistent Qualification Rate Disparities and Interventions](https://mlsanity.com/paper/2111.01201/) (2021)
- [What-is and How-to for Fairness in Machine Learning: A Survey, Reflection, and Perspective](https://mlsanity.com/paper/2206.04101/) (2022)
- [Algorithmic Fairness in Business Analytics: Directions for Research and Practice](https://mlsanity.com/paper/2207.10991/) (2022)
- [Runtime Monitoring of Dynamic Fairness Properties](https://mlsanity.com/paper/2305.04699/) (2023)
- [Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML](https://mlsanity.com/paper/2303.08485/) (2023)
- [The effect of differential victim crime reporting on predictive policing systems](https://mlsanity.com/paper/2102.00128/) (2021)
- [Towards Return Parity in Markov Decision Processes](https://mlsanity.com/paper/2111.10476/) (2021)
- [A Classification of Feedback Loops and Their Relation to Biases in Automated Decision-Making Systems](https://mlsanity.com/paper/2305.06055/) (2023)
- [Fair When Trained, Unfair When Deployed: Observable Fairness Measures are Unstable in Performative Prediction Settings](https://mlsanity.com/paper/2202.05049/) (2022)
- [Non-linear Welfare-Aware Strategic Learning](https://mlsanity.com/paper/2405.01810/) (2024)
- [Designing Closed Human-in-the-loop Deferral Pipelines](https://mlsanity.com/paper/2202.04718/) (2022)
- [Addressing Polarization and Unfairness in Performative Prediction](https://mlsanity.com/paper/2406.16756/) (2024)
- [Recommendation Fairness: From Static to Dynamic](https://mlsanity.com/paper/2109.03150/) (2021)
- [Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutions](https://mlsanity.com/paper/2405.08027/) (2024)
- [Monitoring Algorithmic Fairness](https://mlsanity.com/paper/2305.15979/) (2023)
- [Tier Balancing: Towards Dynamic Fairness over Underlying Causal Factors](https://mlsanity.com/paper/2301.08987/) (2023)

## Abstract (arXiv)

Predictive policing systems are increasingly used to determine how to allocate police across a city in order to best prevent crime. Discovered crime data (e.g., arrest counts) are used to help update the model, and the process is repeated. Such systems have been empirically shown to be susceptible to runaway feedback loops, where police are repeatedly sent back to the same neighborhoods regardless of the true crime rate. In response, we develop a mathematical model of predictive policing that proves why this feedback loop occurs, show empirically that this model exhibits such problems, and demonstrate how to change the inputs to a predictive policing system (in a black-box manner) so the runaway feedback loop does not occur, allowing the true crime rate to be learned. Our results are quantitative: we can establish a link (in our model) between the degree to which runaway feedback causes problems and the disparity in crime rates between areas. Moreover, we can also demonstrate the way in which reported incidents of crime (those reported by residents) and discovered incidents of crime (i.e. those directly observed by police officers dispatched as a result of the predictive policing algorithm) interact: in brief, while reported incidents can attenuate the degree of runaway feedback, they cannot entirely remove it without the interventions we suggest.