mlsanityData to 13 Oct 2026

When combinations of humans and AI are useful: A systematic review and meta-analysis

arXiv 2405.06087 ↗First posted 9 May 2024cs.HC cs.AI cs.CYcited by 105

Not on the map. Why?

Abstract (arXiv)

Inspired by the increasing use of AI to augment humans, researchers have studied human-AI systems involving different tasks, systems, and populations. Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone. Here, we addressed this question by conducting a meta-analysis of over 100 recent experimental studies reporting over 300 effect sizes. First, we found that, on average, human-AI combinations performed significantly worse than the best of humans or AI alone. Second, we found performance losses in tasks that involved making decisions and significantly greater gains in tasks that involved creating content. Finally, when humans outperformed AI alone, we found performance gains in the combination, but when the AI outperformed humans alone we found losses. These findings highlight the heterogeneity of the effects of human-AI collaboration and point to promising avenues for improving human-AI systems.

Key facts. Cited by 105 papers in the map (data through 13 Oct 2026). Builds on 22 of its 77 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

22 of 77 references.

References · 22

  1. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
  2. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  3. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  4. 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 148
  5. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
  6. 2020Learning to Complement Humanscited by 124
  7. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  8. 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
  9. 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86
  10. 2023Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanationscited by 73
  11. 2020Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-makingcited by 59
  12. 2021Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Makingcited by 58
10 more
  1. 2021Deepfake Detection by Human Crowds, Machines, and Machine-informed Crowdscited by 58
  2. 2022Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairnesscited by 45
  3. 2023Improving Human-AI Collaboration With Descriptions of AI Behaviorcited by 36
  4. 2021Role of Human-AI Interaction in Selective Predictioncited by 34
  5. 2022A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Makingcited by 29
  6. 2022A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementaritycited by 23
  7. 2023Effective Human-AI Teams via Learned Natural Language Rules and Onboardingcited by 20
  8. 2022Learning When to Advise Human Decision Makerscited by 18
  9. 2023"An Adapt-or-Die Type of Situation": Perception, Adoption, and Use of Text-To-Image-Generation AI by Game Industry Professionalscited by 11
  10. 2022A Test for Evaluating Performance in Human-Computer Systemscited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 7; 2025: 34; 2026: 64.

105 papers.

Sort:

By area:

  1. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 61
  2. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 41
  3. 2024Divergent Creativity in Humans and Large Language Modelscited by 29
  4. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 28
  5. 2025SciSciGPT: Advancing Human-AI Collaboration in the Science of Sciencecited by 26
  6. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 18
  7. 2025Collaborating with AI Agents: Field Experiments on Teamwork, Productivity, and Performancecited by 14
  8. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 12
  9. 2024A No Free Lunch Theorem for Human-AI Collaborationcited by 8
  10. 2025When Should We Orchestrate Multiple Agents?cited by 7
  11. 2025Human-AI Collaborative Uncertainty Quantificationcited by 7
  12. 2025Explaining and Improving Information Complementarities in Multi-Agent Decision-makingcited by 6
38 more
  1. 2025Advancing AI Negotiations: A Large-Scale Autonomous Negotiation Competitioncited by 6
  2. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  3. 2025Everything is Context: Agentic File System Abstraction for Context Engineeringcited by 6
  4. 2026AI Agents for Inventory Control: Human-LLM-OR Complementaritycited by 6
  5. 2025To Ask or Not to Ask: Learning to Require Human Feedbackcited by 5
  6. 2025Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaborationcited by 4
  7. 2025MetaMuse: Algorithm Generation via Creative Ideationcited by 4
  8. 2026Interactive Narrative Analytics: Bridging Computational Narrative Extraction and Human Sensemakingcited by 4
  9. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
  10. 2026The Augmentation Trap: AI Productivity and the Cost of Cognitive Offloadingcited by 4
  11. 2025Sustainability via LLM Right-sizingcited by 3
  12. 2025DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfacescited by 3
  13. 2025Human-AI Complementarity: A Goal for Amplified Oversightcited by 3
  14. 2026More Human or More AI? Visualizing Human-AI Collaboration Disclosures in Journalistic News Productioncited by 3
  15. 2026Multi-Round Human-AI Collaboration with User-Specified Requirementscited by 3
  16. 2026Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Upliftcited by 3
  17. 2026(Human) Attention Is (Still) All You Need: Human oversight makes AI-assisted social science reliablecited by 3
  18. 2024Aligning Generalisation Between Humans and Machinescited by 2
  19. 2025Revisiting Rogers' Paradox in the Context of Human-AI Interactioncited by 2
  20. 2025Reliable agent engineering should integrate machine-compatible organizational principlescited by 2
  21. 2026Human Tool: An MCP-Style Framework for Human-Agent Collaborationcited by 2
  22. 2026Human-AI Productivity Paradoxes: Modeling the Interplay of Skill, Effort, and AI Assistancecited by 2
  23. 2026Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systemscited by 2
  24. 2024Confidence-weighted integration of human and machine judgments for superior decision-makingcited by 1
  25. 2025A Comparison of Human and Machine Learning Errors in Face Recognitioncited by 1
  26. 2025Evaluating Trust in AI, Human, and Co-produced Feedback Among Undergraduate Studentscited by 1
  27. 2025A Mathematical Framework for AI-Human Integration in Workcited by 1
  28. 2025Explanations are a Means to an End: Decision Theoretic Explanation Evaluationcited by 1
  29. 2025Making LLMs Reliable When It Matters Most: A Five-Layer Architecture for High-Stakes Decisionscited by 1
  30. 2025From Augmentation to Symbiosis: A Review of Human-AI Collaboration Frameworks, Performance, and Perilscited by 1
  31. 2026A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarkingcited by 1
  32. 2026Toward Trustworthy Evaluation of Sustainability Rating Methodologies: A Human-AI Collaborative Framework for Benchmark Dataset Constructioncited by 1
  33. 2026ComplLLM: Fine-tuning LLMs to Discover Complementary Signals for Decision-makingcited by 1
  34. 2026Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Studycited by 1
  35. 2026Cognitive Amplification vs Cognitive Delegation in Human-AI Systems: A Metric Frameworkcited by 1
  36. 2026Effects of Generative AI Errors on User Reliance Across Task Difficultycited by 1
  37. 2026Framing, Judging, Steering: An Assessable Competency Model for Teach-ing Students to Reason With Generative AIcited by 1
  38. 2026Strategic Decision Support for AI Agentscited by 1

50 most cited of 105 All 105 (JSON)