# When combinations of humans and AI are useful: A systematic review and meta-analysis

arXiv: https://arxiv.org/abs/2405.06087 · HTML: https://mlsanity.com/paper/2405.06087/

## Key facts

- Cited by 106 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 77 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 22 of 22)

- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)
- [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making](https://mlsanity.com/paper/2102.09692/) (2021)
- [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022)
- ["Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans](https://mlsanity.com/paper/2001.05871/) (2020)
- [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023)
- [Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making](https://mlsanity.com/paper/2010.07938/) (2020)
- [Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making](https://mlsanity.com/paper/2101.05303/) (2021)
- [Deepfake Detection by Human Crowds, Machines, and Machine-informed Crowds](https://mlsanity.com/paper/2105.06496/) (2021)
- [Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness](https://mlsanity.com/paper/2202.08821/) (2022)
- [Improving Human-AI Collaboration With Descriptions of AI Behavior](https://mlsanity.com/paper/2301.06937/) (2023)
- [Role of Human-AI Interaction in Selective Prediction](https://mlsanity.com/paper/2112.06751/) (2021)
- [A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making](https://mlsanity.com/paper/2205.05126/) (2022)
- [A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementarity](https://mlsanity.com/paper/2204.10806/) (2022)
- [Effective Human-AI Teams via Learned Natural Language Rules and Onboarding](https://mlsanity.com/paper/2311.01007/) (2023)
- [Learning When to Advise Human Decision Makers](https://mlsanity.com/paper/2209.13578/) (2022)
- ["An Adapt-or-Die Type of Situation": Perception, Adoption, and Use of Text-To-Image-Generation AI by Game Industry Professionals](https://mlsanity.com/paper/2302.12601/) (2023)
- [A Test for Evaluating Performance in Human-Computer Systems](https://mlsanity.com/paper/2206.12390/) (2022)

## Built on by (most cited first, 30 of 106)

- [Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration](https://mlsanity.com/paper/2412.15701/) (2024)
- [Magentic-UI: Towards Human-in-the-loop Agentic Systems](https://mlsanity.com/paper/2507.22358/) (2025)
- [Divergent Creativity in Humans and Large Language Models](https://mlsanity.com/paper/2405.13012/) (2024)
- [This human study did not involve human subjects: Validating LLM simulations as behavioral evidence](https://mlsanity.com/paper/2602.15785/) (2026)
- [SciSciGPT: Advancing Human-AI Collaboration in the Science of Science](https://mlsanity.com/paper/2504.05559/) (2025)
- [How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations](https://mlsanity.com/paper/2510.22780/) (2025)
- [Collaborating with AI Agents: Field Experiments on Teamwork, Productivity, and Performance](https://mlsanity.com/paper/2503.18238/) (2025)
- [CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding](https://mlsanity.com/paper/2512.04111/) (2025)
- [A No Free Lunch Theorem for Human-AI Collaboration](https://mlsanity.com/paper/2411.15230/) (2024)
- [AI Agents for Inventory Control: Human-LLM-OR Complementarity](https://mlsanity.com/paper/2602.12631/) (2026)
- [When Should We Orchestrate Multiple Agents?](https://mlsanity.com/paper/2503.13577/) (2025)
- [Human-AI Collaborative Uncertainty Quantification](https://mlsanity.com/paper/2510.23476/) (2025)
- [Explaining and Improving Information Complementarities in Multi-Agent Decision-making](https://mlsanity.com/paper/2502.06152/) (2025)
- [Advancing AI Negotiations: A Large-Scale Autonomous Negotiation Competition](https://mlsanity.com/paper/2503.06416/) (2025)
- [Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variability](https://mlsanity.com/paper/2504.12320/) (2025)
- [To Ask or Not to Ask: Learning to Require Human Feedback](https://mlsanity.com/paper/2510.08314/) (2025)
- [Everything is Context: Agentic File System Abstraction for Context Engineering](https://mlsanity.com/paper/2512.05470/) (2025)
- [DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces](https://mlsanity.com/paper/2509.13444/) (2025)
- [Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration](https://mlsanity.com/paper/2509.18008/) (2025)
- [MetaMuse: Algorithm Generation via Creative Ideation](https://mlsanity.com/paper/2510.03851/) (2025)
- [Interactive Narrative Analytics: Bridging Computational Narrative Extraction and Human Sensemaking](https://mlsanity.com/paper/2601.11459/) (2026)
- [Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents](https://mlsanity.com/paper/2602.01405/) (2026)
- [The Augmentation Trap: AI Productivity and the Cost of Cognitive Offloading](https://mlsanity.com/paper/2604.03501/) (2026)
- [Sustainability via LLM Right-sizing](https://mlsanity.com/paper/2504.13217/) (2025)
- [Human-AI Complementarity: A Goal for Amplified Oversight](https://mlsanity.com/paper/2510.26518/) (2025)
- [More Human or More AI? Visualizing Human-AI Collaboration Disclosures in Journalistic News Production](https://mlsanity.com/paper/2601.11072/) (2026)
- [Multi-Round Human-AI Collaboration with User-Specified Requirements](https://mlsanity.com/paper/2602.17646/) (2026)
- [Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift](https://mlsanity.com/paper/2603.26676/) (2026)
- [(Human) Attention Is (Still) All You Need: Human oversight makes AI-assisted social science reliable](https://mlsanity.com/paper/2606.12848/) (2026)
- [Aligning Generalisation Between Humans and Machines](https://mlsanity.com/paper/2411.15626/) (2024)

## Abstract (arXiv)

Inspired by the increasing use of AI to augment humans, researchers have studied human-AI systems involving different tasks, systems, and populations. Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone. Here, we addressed this question by conducting a meta-analysis of over 100 recent experimental studies reporting over 300 effect sizes. First, we found that, on average, human-AI combinations performed significantly worse than the best of humans or AI alone. Second, we found performance losses in tasks that involved making decisions and significantly greater gains in tasks that involved creating content. Finally, when humans outperformed AI alone, we found performance gains in the combination, but when the AI outperformed humans alone we found losses. These findings highlight the heterogeneity of the effects of human-AI collaboration and point to promising avenues for improving human-AI systems.