# ProPublica's COMPAS Data Revisited

arXiv: https://arxiv.org/abs/1906.04711 · HTML: https://mlsanity.com/paper/1906.04711/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 24 of 24)

- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Benchmarking Distribution Shift in Tabular Data with TableShift](https://mlsanity.com/paper/2312.07577/) (2023)
- [Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluation](https://mlsanity.com/paper/2211.13358/) (2022)
- [Unprocessing Seven Years of Algorithmic Fairness](https://mlsanity.com/paper/2306.07261/) (2023)
- [When is Multicalibration Post-Processing Necessary?](https://mlsanity.com/paper/2406.06487/) (2024)
- [Subgroup Robustness Grows On Trees: An Empirical Baseline Investigation](https://mlsanity.com/paper/2211.12703/) (2022)
- [Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approach](https://mlsanity.com/paper/2109.08549/) (2021)
- [FFPDG: Fast, Fair and Private Data Generation](https://mlsanity.com/paper/2307.00161/) (2023)
- [The Utility of "Even if..." Semifactual Explanation to Optimise Positive Outcomes](https://mlsanity.com/paper/2310.18937/) (2023)
- [Assessment of Differentially Private Synthetic Data for Utility and Fairness in End-to-End Machine Learning Pipelines for Tabular Data](https://mlsanity.com/paper/2310.19250/) (2023)
- [An Analysis of the Deployment of Models Trained on Private Tabular Synthetic Data: Unexpected Surprises](https://mlsanity.com/paper/2106.10241/) (2021)
- [Is your model predicting the past?](https://mlsanity.com/paper/2206.11673/) (2022)
- [Conformalized Selective Regression](https://mlsanity.com/paper/2402.16300/) (2024)
- [PraFFL: A Preference-Aware Scheme in Fair Federated Learning](https://mlsanity.com/paper/2404.08973/) (2024)
- [Investigating Imperceptibility of Adversarial Attacks on Tabular Data: An Empirical Analysis](https://mlsanity.com/paper/2407.11463/) (2024)
- [How Robust is your Fair Model? Exploring the Robustness of Diverse Fairness Strategies](https://mlsanity.com/paper/2207.04581/) (2022)
- [Deontic Temporal Logic for Formal Verification of AI Ethics](https://mlsanity.com/paper/2501.05765/) (2025)
- [Bias Mitigation Framework for Intersectional Subgroups in Neural Networks](https://mlsanity.com/paper/2212.13014/) (2022)
- [Provable Detection of Propagating Sampling Bias in Prediction Models](https://mlsanity.com/paper/2302.06752/) (2023)
- [Asymptotically Fair Participation in Machine Learning Models: an Optimal Control Perspective](https://mlsanity.com/paper/2311.10223/) (2023)
- [Towards Harmless Rawlsian Fairness Regardless of Demographic Prior](https://mlsanity.com/paper/2411.02467/) (2024)
- [Alpha and Prejudice: Improving α-sized Worst-case Fairness via Intrinsic Reweighting](https://mlsanity.com/paper/2411.03068/) (2024)
- [Learning Heterogeneous Performance-Fairness Trade-offs in Federated Learning](https://mlsanity.com/paper/2504.21775/) (2025)
- [Locally Adaptive Multi-Objective Learning](https://mlsanity.com/paper/2602.14952/) (2026)

## Abstract (arXiv)

I examine the COMPAS recidivism risk score and criminal history data collected by ProPublica in 2016 that fueled intense debate and research in the nascent field of 'algorithmic fairness'. ProPublica's COMPAS data is used in an increasing number of studies to test various definitions of algorithmic fairness. This paper takes a closer look at the actual datasets put together by ProPublica. In particular, the sub-datasets built to study the likelihood of recidivism within two years of a defendant's original COMPAS survey screening date. I take a new yet simple approach to visualize these data, by analyzing the distribution of defendants across COMPAS screening dates. I find that ProPublica made an important data processing error when it created these datasets, failing to implement a two-year sample cutoff rule for recidivists in such datasets (whereas it implemented a two-year sample cutoff rule for non-recidivists). When I implement a simple two-year COMPAS screen date cutoff rule for recidivists, I estimate that in the two-year general recidivism dataset ProPublica kept over 40% more recidivists than it should have. This fundamental problem in dataset construction affects some statistics more than others. It obviously has a substantial impact on the recidivism rate; artificially inflating it. For the two-year general recidivism dataset created by ProPublica, the two-year recidivism rate is 45.1%, whereas, with the simple COMPAS screen date cutoff correction I implement, it is 36.2%. Thus, the two-year recidivism rate in ProPublica's dataset is inflated by over 24%. This also affects the positive and negative predictive values. On the other hand, this data processing error has little impact on some of the other key statistical measures, which are less susceptible to changes in the relative share of recidivists, such as the false positive and false negative rates, and the overall accuracy.