# The age of secrecy and unfairness in recidivism prediction

arXiv: https://arxiv.org/abs/1811.00731 · HTML: https://mlsanity.com/paper/1811.00731/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2018.

## Built on by (most cited first, 30 of 53)

- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Counterfactual Explanations Can Be Manipulated](https://mlsanity.com/paper/2106.02666/) (2021)
- [Characterizing Fairness Over the Set of Good Models Under Selective Labels](https://mlsanity.com/paper/2101.00352/) (2021)
- [Amazing Things Come From Having Many Good Models](https://mlsanity.com/paper/2407.04846/) (2024)
- [The Who in XAI: How AI Background Shapes Perceptions of AI Explanations](https://mlsanity.com/paper/2107.13509/) (2021)
- [The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail](https://mlsanity.com/paper/2110.08300/) (2021)
- [Explainability Pitfalls: Beyond Dark Patterns in Explainable AI](https://mlsanity.com/paper/2109.12480/) (2021)
- [Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI](https://mlsanity.com/paper/2302.00799/) (2023)
- [Constrained Monotonic Neural Networks](https://mlsanity.com/paper/2205.11775/) (2022)
- [A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms](https://mlsanity.com/paper/2206.14983/) (2022)
- [Assessing Fairness in the Presence of Missing Data](https://mlsanity.com/paper/2112.04899/) (2021)
- [Rational Shapley Values](https://mlsanity.com/paper/2106.10191/) (2021)
- [Subgroup Robustness Grows On Trees: An Empirical Baseline Investigation](https://mlsanity.com/paper/2211.12703/) (2022)
- [A Holistic Approach to Interpretability in Financial Lending: Models, Visualizations, and Summary-Explanations](https://mlsanity.com/paper/2106.02605/) (2021)
- [FADE: FAir Double Ensemble Learning for Observable and Counterfactual Outcomes](https://mlsanity.com/paper/2109.00173/) (2021)
- [Partial Order in Chaos: Consensus on Feature Attributions in the Rashomon Set](https://mlsanity.com/paper/2110.13369/) (2021)
- [Safe Policy Learning through Extrapolation: Application to Pre-trial Risk Assessment](https://mlsanity.com/paper/2109.11679/) (2021)
- [AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development](https://mlsanity.com/paper/2502.18682/) (2025)
- [Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm](https://mlsanity.com/paper/2206.02976/) (2022)
- [Interpretability with full complexity by constraining feature information](https://mlsanity.com/paper/2211.17264/) (2022)
- [FederatedTrust: A Solution for Trustworthy Federated Learning](https://mlsanity.com/paper/2302.09844/) (2023)
- [10 Years of Fair Representations: Challenges and Opportunities](https://mlsanity.com/paper/2407.03834/) (2024)
- [User Driven Model Adjustment via Boolean Rule Explanations](https://mlsanity.com/paper/2203.15071/) (2022)
- [Variable importance without impossible data](https://mlsanity.com/paper/2205.15750/) (2022)
- [Is your model predicting the past?](https://mlsanity.com/paper/2206.11673/) (2022)
- [Exploring the cloud of feature interaction scores in a Rashomon set](https://mlsanity.com/paper/2305.10181/) (2023)
- [Uncovering Unique Concept Vectors through Latent Space Decomposition](https://mlsanity.com/paper/2307.06913/) (2023)
- [Doing Right by Not Doing Wrong in Human-Robot Collaboration](https://mlsanity.com/paper/2202.02654/) (2022)
- [Fast Feature Selection with Fairness Constraints](https://mlsanity.com/paper/2202.13718/) (2022)
- [Invariant Representations with Stochastically Quantized Neural Networks](https://mlsanity.com/paper/2208.02656/) (2022)

## Abstract (arXiv)

In our current society, secret algorithms make important decisions about individuals. There has been substantial discussion about whether these algorithms are unfair to groups of individuals. While noble, this pursuit is complex and ultimately stagnating because there is no clear definition of fairness and competing definitions are largely incompatible. We argue that the focus on the question of fairness is misplaced, as these algorithms fail to meet a more important and yet readily obtainable goal: transparency. As a result, creators of secret algorithms can provide incomplete or misleading descriptions about how their models work, and various other kinds of errors can easily go unnoticed. By partially reverse engineering the COMPAS algorithm -- a recidivism-risk scoring algorithm used throughout the criminal justice system -- we show that it does not seem to depend linearly on the defendant's age, despite statements to the contrary by the algorithm's creator. Furthermore, by subtracting from COMPAS its (hypothesized) nonlinear age component, we show that COMPAS does not necessarily depend on race, contradicting ProPublica's analysis, which assumed linearity in age. In other words, faulty assumptions about a proprietary algorithm lead to faulty conclusions that go unchecked without careful reverse engineering. Were the algorithm transparent in the first place, this would likely not have occurred. The most important result in this work is that we find that there are many defendants with low risk score but long criminal histories, suggesting that data inconsistencies occur frequently in criminal justice databases. We argue that transparency satisfies a different notion of procedural fairness by providing both the defendants and the public with the opportunity to scrutinize the methodology and calculations behind risk scores for recidivism.