# Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government Contexts

arXiv: https://arxiv.org/abs/2012.05370 · HTML: https://mlsanity.com/paper/2012.05370/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 22 of 22)

- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant](https://mlsanity.com/paper/2502.01390/) (2025)
- [Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems](https://mlsanity.com/paper/2301.11333/) (2023)
- [Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithm](https://mlsanity.com/paper/2207.02942/) (2022)
- [Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant](https://mlsanity.com/paper/2501.17546/) (2025)
- [Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making](https://mlsanity.com/paper/2302.06503/) (2023)
- [A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementarity](https://mlsanity.com/paper/2204.10806/) (2022)
- [Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions](https://mlsanity.com/paper/2302.07248/) (2023)
- [Leveraging Expert Consistency to Improve Algorithmic Decision Support](https://mlsanity.com/paper/2101.09648/) (2021)
- [Algorithmic Assistance with Recommendation-Dependent Preferences](https://mlsanity.com/paper/2208.07626/) (2022)
- [On (assessing) the fairness of risk score models](https://mlsanity.com/paper/2302.08851/) (2023)
- [To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems](https://mlsanity.com/paper/2409.14377/) (2024)
- [Escaping the Impossibility of Fairness: From Formal to Substantive Algorithmic Fairness](https://mlsanity.com/paper/2107.04642/) (2021)
- [Do Prevalent Bias Metrics Capture Allocational Harms from LLMs?](https://mlsanity.com/paper/2408.01285/) (2024)
- [Fine-Grained Appropriate Reliance: Human-AI Collaboration with a Multi-Step Transparent Decision Workflow for Complex Task Decomposition](https://mlsanity.com/paper/2501.10909/) (2025)
- [Learning to Advise Humans in High-Stakes Settings](https://mlsanity.com/paper/2210.12849/) (2022)
- [Resume-ing Control: (Mis)Perceptions of Agency Around GenAI Use in Recruiting Workflows](https://mlsanity.com/paper/2604.26851/) (2026)
- [Human Response to an AI-Based Decision Support System: A User Study on the Effects of Accuracy and Bias](https://mlsanity.com/paper/2203.15514/) (2022)
- [Mind the Gap: Optimal and Equitable Encouragement Policies](https://mlsanity.com/paper/2309.07176/) (2023)
- [The Value of AI Advice: Personalized and Value-Maximizing AI Advisors Are Necessary to Reliably Benefit Experts and Organizations](https://mlsanity.com/paper/2412.19530/) (2024)
- [The Illusion of Opting in AI-Mediated Consequential Decisions](https://mlsanity.com/paper/2605.28210/) (2026)
- [A Technical Typology of AI Systems in Public Administration](https://mlsanity.com/paper/2606.31755/) (2026)

## Abstract (arXiv)

Governments are increasingly turning to algorithmic risk assessments when making important decisions, such as whether to release criminal defendants before trial. Policymakers assert that providing public servants with algorithmic advice will improve human risk predictions and thereby lead to better (e.g., fairer) decisions. Yet because many policy decisions require balancing risk-reduction with competing goals, improving the accuracy of predictions may not necessarily improve the quality of decisions. If risk assessments make people more attentive to reducing risk at the expense of other values, these algorithms would diminish the implementation of public policy even as they lead to more accurate predictions. Through an experiment with 2,140 lay participants simulating two high-stakes government contexts, we provide the first direct evidence that risk assessments can systematically alter how people factor risk into their decisions. These shifts counteracted the potential benefits of improved prediction accuracy. In the pretrial setting of our experiment, the risk assessment made participants more sensitive to increases in perceived risk; this shift increased the racial disparity in pretrial detention by 1.9%. In the government loans setting of our experiment, the risk assessment made participants more risk-averse; this shift reduced government aid by 8.3%. These results demonstrate the potential limits and harms of attempts to improve public policy by incorporating predictive algorithms into multifaceted policy decisions. If these observed behaviors occur in practice, presenting risk assessments to public servants would generate unexpected and unjust shifts in public policy without being subject to democratic deliberation or oversight.