# A Test for Evaluating Performance in Human-Computer Systems

arXiv: https://arxiv.org/abs/2206.12390 · HTML: https://mlsanity.com/paper/2206.12390/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 81 references that are in the map.
- First posted on arXiv 24 Jun 2022.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making](https://mlsanity.com/paper/2102.09692/) (2021)
- [Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making](https://mlsanity.com/paper/2101.05303/) (2021)
- [How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations](https://mlsanity.com/paper/2101.08758/) (2021)
- [Deepfake Detection by Human Crowds, Machines, and Machine-informed Crowds](https://mlsanity.com/paper/2105.06496/) (2021)
- [The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems](https://mlsanity.com/paper/2105.03354/) (2021)
- [Role of Human-AI Interaction in Selective Prediction](https://mlsanity.com/paper/2112.06751/) (2021)
- [Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative](https://mlsanity.com/paper/2103.04725/) (2021)
- [Increasing the Speed and Accuracy of Data LabelingThrough an AI Assisted Interface](https://mlsanity.com/paper/2104.04122/) (2021)
- [Designing for human-AI complementarity in K-12 education](https://mlsanity.com/paper/2104.01266/) (2021)
- [Automatic Discovery of Interpretable Planning Strategies](https://mlsanity.com/paper/2005.11730/) (2020)
- [Human-Level Intelligence or Animal-Like Abilities?](https://mlsanity.com/paper/1707.04327/) (2017)
- [Addressing Cognitive Biases in Augmented Business Decision Systems](https://mlsanity.com/paper/2009.08127/) (2020)

## Built on by (most cited first, 8 of 8)

- [When combinations of humans and AI are useful: A systematic review and meta-analysis](https://mlsanity.com/paper/2405.06087/) (2024)
- [Toward General Design Principles for Generative AI Applications](https://mlsanity.com/paper/2301.05578/) (2023)
- [Shifting Work Patterns with Generative AI](https://mlsanity.com/paper/2504.11436/) (2025)
- [Early Impacts of M365 Copilot](https://mlsanity.com/paper/2504.11443/) (2025)
- [Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift](https://mlsanity.com/paper/2603.26676/) (2026)
- [Rethinking Algorithmic Fairness for Human-AI Collaboration](https://mlsanity.com/paper/2310.03647/) (2023)
- [A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health](https://mlsanity.com/paper/2411.02594/) (2024)
- [Useful for Exploration, Risky for Precision: Evaluating AI Tools in Academic Research](https://mlsanity.com/paper/2605.10125/) (2026)

## Abstract (arXiv)

The Turing test for comparing computer performance to that of humans is well known, but, surprisingly, there is no widely used test for comparing how much better human-computer systems perform relative to humans alone, computers alone, or other baselines. Here, we show how to perform such a test using the ratio of means as a measure of effect size. Then we demonstrate the use of this test in three ways. First, in an analysis of 79 recently published experimental results, we find that, surprisingly, over half of the studies find a decrease in performance, the mean and median ratios of performance improvement are both approximately 1 (corresponding to no improvement at all), and the maximum ratio is 1.36 (a 36% improvement). Second, we experimentally investigate whether a higher performance improvement ratio is obtained when 100 human programmers generate software using GPT-3, a massive, state-of-the-art AI system. In this case, we find a speed improvement ratio of 1.27 (a 27% improvement). Finally, we find that 50 human non-programmers using GPT-3 can perform the task about as well as--and less expensively than--the human programmers. In this case, neither the non-programmers nor the computer would have been able to perform the task alone, so this is an example of a very strong form of human-computer synergy.