# Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment

arXiv: https://arxiv.org/abs/2012.02845 · HTML: https://mlsanity.com/paper/2012.02845/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2020.

## Built on by (most cited first, 18 of 18)

- [Causal Conceptions of Fairness and their Consequences](https://mlsanity.com/paper/2207.05302/) (2022)
- [Algorithmic Assistance with Recommendation-Dependent Preferences](https://mlsanity.com/paper/2208.07626/) (2022)
- [On the Fairness of Machine-Assisted Human Decisions](https://mlsanity.com/paper/2110.15310/) (2021)
- [Safe Policy Learning through Extrapolation: Application to Pre-trial Risk Assessment](https://mlsanity.com/paper/2109.11679/) (2021)
- [Does AI help humans make better decisions? A statistical evaluation framework for experimental and observational studies](https://mlsanity.com/paper/2403.12108/) (2024)
- [Bayesian Safe Policy Learning with Chance Constrained Optimization: Application to Military Security Assessment during the Vietnam War](https://mlsanity.com/paper/2307.08840/) (2023)
- [Designing Algorithmic Recommendations to Achieve Human-AI Complementarity](https://mlsanity.com/paper/2405.01484/) (2024)
- [Causal Fairness Assessment of Treatment Allocation with Electronic Health Records](https://mlsanity.com/paper/2211.11183/) (2022)
- [Persuasion, Delegation, and Private Information in Algorithm-Assisted Decisions](https://mlsanity.com/paper/2402.09384/) (2024)
- [Designing Equitable Algorithms](https://mlsanity.com/paper/2302.09157/) (2023)
- [Decision-aid or Controller? Steering Human Decision Makers with Algorithms](https://mlsanity.com/paper/2303.13712/) (2023)
- [Statistical Decision Theory with Counterfactual Loss](https://mlsanity.com/paper/2505.08908/) (2025)
- [Mind the Gap: Optimal and Equitable Encouragement Policies](https://mlsanity.com/paper/2309.07176/) (2023)
- [Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning](https://mlsanity.com/paper/2310.07973/) (2023)
- [Peer-induced Fairness: A Causal Approach for Algorithmic Fairness Auditing](https://mlsanity.com/paper/2408.02558/) (2024)
- [2-Step Agent: A Framework for the Interaction of a Decision Maker with AI Decision Support](https://mlsanity.com/paper/2602.21889/) (2026)
- [Man and machine: artificial intelligence and judicial decision making](https://mlsanity.com/paper/2603.19042/) (2026)
- [Set-Valued Policy Learning](https://mlsanity.com/paper/2605.19830/) (2026)

## Abstract (arXiv)

Despite an increasing reliance on fully-automated algorithmic decision-making in our day-to-day lives, human beings still make highly consequential decisions. As frequently seen in business, healthcare, and public policy, recommendations produced by algorithms are provided to human decision-makers to guide their decisions. While there exists a fast-growing literature evaluating the bias and fairness of such algorithmic recommendations, an overlooked question is whether they help humans make better decisions. We develop a statistical methodology for experimentally evaluating the causal impacts of algorithmic recommendations on human decisions. We also show how to examine whether algorithmic recommendations improve the fairness of human decisions and derive the optimal decision rules under various settings. We apply the proposed methodology to preliminary data from the first-ever randomized controlled trial that evaluates the pretrial Public Safety Assessment (PSA) in the criminal justice system. A goal of the PSA is to help judges decide which arrested individuals should be released. On the basis of the preliminary data available, we find that providing the PSA to the judge has little overall impact on the judge's decisions and subsequent arrestee behavior. However, our analysis yields some potentially suggestive evidence that the PSA may help avoid unnecessarily harsh decisions for female arrestees regardless of their risk levels while it encourages the judge to make stricter decisions for male arrestees who are deemed to be risky. In terms of fairness, the PSA appears to increase the gender bias against males while having little effect on any existing racial differences in judges' decision. Finally, we find that the PSA's recommendations might be unnecessarily severe unless the cost of a new crime is sufficiently high.