# Does AI help humans make better decisions? A statistical evaluation framework for experimental and observational studies

arXiv: https://arxiv.org/abs/2403.12108 · HTML: https://mlsanity.com/paper/2403.12108/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 32 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 5 of 5)

- [Fairness in Machine Learning](https://mlsanity.com/paper/2012.15816/) (2020)
- ['It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions](https://mlsanity.com/paper/1801.10408/) (2018)
- [Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment](https://mlsanity.com/paper/2012.02845/) (2020)
- [Policy Learning with Asymmetric Counterfactual Utilities](https://mlsanity.com/paper/2206.10479/) (2022)
- [Heterogeneity in Algorithm-Assisted Decision-Making: A Case Study in Child Abuse Hotline Screening](https://mlsanity.com/paper/2204.05478/) (2022)

## Built on by (most cited first, 7 of 7)

- [Explaining and Improving Information Complementarities in Multi-Agent Decision-making](https://mlsanity.com/paper/2502.06152/) (2025)
- [Statistical Decision Theory with Counterfactual Loss](https://mlsanity.com/paper/2505.08908/) (2025)
- [Epitome: Pioneering an Experimental Platform for AI-Social Science Integration](https://mlsanity.com/paper/2507.01061/) (2025)
- [Man and machine: artificial intelligence and judicial decision making](https://mlsanity.com/paper/2603.19042/) (2026)
- [Set-Valued Policy Learning](https://mlsanity.com/paper/2605.19830/) (2026)
- [ForkSCOPE: Charting the Agentic Garden of Forking Paths](https://mlsanity.com/paper/2609.12438/) (2026)
- [Structural Alignment for Reliable Industrial AI: Bridging Physical Reality, Data, Models, and Human Intent](https://mlsanity.com/paper/2609.35400/) (2026)

## Abstract (arXiv)

The use of Artificial Intelligence (AI), or more generally data-driven algorithms, has become ubiquitous in today's society. Yet, in many cases and especially when stakes are high, humans still make final decisions. The critical question, therefore, is whether AI helps humans make better decisions compared to a human-alone or AI-alone system. We introduce a new methodological framework to empirically answer this question with a minimal set of assumptions. We measure a decision maker's ability to make correct decisions using standard classification metrics based on the baseline potential outcome. We consider a single-blinded and unconfounded treatment assignment, where the provision of AI-generated recommendations is assumed to be randomized across cases with humans making final decisions. Under this study design, we show how to compare the performance of three alternative decision-making systems--human-alone, human-with-AI, and AI-alone. Importantly, the AI-alone system includes any individualized treatment assignment, including those that are not used in the original study. We also show when AI recommendations should be provided to a human-decision maker, and when one should follow such recommendations. We apply the proposed methodology to our own randomized controlled trial evaluating a pretrial risk assessment instrument. We find that the risk assessment recommendations do not improve the classification accuracy of a judge's decision to impose cash bail. Furthermore, we find that replacing a human judge with algorithms--the risk assessment score and a large language model in particular--leads to a worse classification performance.