mlsanityData to 7 Oct 2026

Evaluating Human-AI Collaboration: A Review and Methodological Framework

arXiv 2407.19098 ↗First posted 9 Jul 2024cs.HC cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

The use of artificial intelligence (AI) in working environments with individuals, known as Human-AI Collaboration (HAIC), has become essential in a variety of domains, boosting decision-making, efficiency, and innovation. Despite HAIC's wide potential, evaluating its effectiveness remains challenging due to the complex interaction of components involved. This paper provides a detailed analysis of existing HAIC evaluation approaches and develops a fresh paradigm for more effectively evaluating these systems. Our framework includes a structured decision tree which assists to select relevant metrics based on distinct HAIC modes (AI-Centric, Human-Centric, and Symbiotic). By including both quantitative and qualitative metrics, the framework seeks to represent HAIC's dynamic and reciprocal nature, enabling the assessment of its impact and success. This framework's practicality can be examined by its application in an array of domains, including manufacturing, healthcare, finance, and education, each of which has unique challenges and requirements. Our hope is that this study will facilitate further research on the systematic evaluation of HAIC in real-world applications.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 41 of its 155 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

41 of 155 references.

References · 41

  1. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  2. 2023Large Language Modelscited by 774
  3. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  4. 2020Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlationcited by 420
  5. 2022Using cognitive psychology to understand GPT-3cited by 389
  6. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  7. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  8. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  9. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  10. 2023Do Large Language Models Know What They Don't Know?cited by 238
  11. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  12. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
29 more
  1. 2023Art and the science of generative AI: A deeper divecited by 93
  2. 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 90
  3. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  4. 2019Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AIcited by 79
  5. 2022Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Supportcited by 68
  6. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  7. 2022Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systemscited by 59
  8. 2024Determinants of LLM-assisted Decision-Makingcited by 55
  9. 2023Supporting Human-AI Collaboration in Auditing LLMs with LLMscited by 46
  10. 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
  11. 2017Cooperating with Machinescited by 39
  12. 2021The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systemscited by 39
  13. 2024Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Makingcited by 38
  14. 2021Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabicited by 37
  15. 2022Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Makingcited by 33
  16. 2023NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systemscited by 27
  17. 2022Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustnesscited by 23
  18. 2021Human-AI Interactions in Public Sector Decision-Making: "Automation Bias" and "Selective Adherence" to Algorithmic Advicecited by 22
  19. 2021Designing for human-AI complementarity in K-12 educationcited by 19
  20. 2022Human-AI Collaboration in Decision-Making: Beyond Learning to Defercited by 16
  21. 2024Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraintscited by 12
  22. 2021Human-AI Symbiosis: A Survey of Current Approachescited by 8
  23. 2022Optimal Behavior Prior: Data-Efficient Human Models for Improved Human-AI Collaborationcited by 8
  24. 2024A Survey on Human-AI Collaboration with Large Foundation Modelscited by 7
  25. 2023Ethical Framework for Harnessing the Power of AI in Healthcare and Beyondcited by 6
  26. 2023Hierarchical Evaluation Framework: Best Practices for Human Evaluationcited by 5
  27. 2022Factors that influence the adoption of human-AI collaboration in clinical decision-makingcited by 4
  28. 2022Team Learning as a Lens for Designing Human-AI Co-Creative Systemscited by 1
  29. 2023Towards Time Sensitive Networking on Smart Cities: Techniques, Challenges, and Solutionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 9.

25 papers.

Sort:

By area:

  1. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 11
  2. 2025Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specificationscited by 5
  3. 2025Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologiescited by 5
  4. 2025Interaction Dynamics as a Reward Signal for LLMscited by 5
  5. 2025When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaborationcited by 4
  6. 2026Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgementcited by 4
  7. 2025Position: Beyond Assistance -- Reimagining LLMs as Ethical and Adaptive Co-Creators in Mental Health Carecited by 3
  8. 2026More Human or More AI? Visualizing Human-AI Collaboration Disclosures in Journalistic News Productioncited by 3
  9. 2024Causal Responsibility Attribution for Human-AI Collaborationcited by 2
  10. 2025A Framework for Situating Innovations, Opportunities, and Challenges in Advancing Vertical Systems with Large AI Modelscited by 2
  11. 2025Classifying Epistemic Relationships in Human-AI Interaction: An Exploratory Approachcited by 2
  12. 2025Collaborative Causal Sensemaking: Closing the Complementarity Gap in Human-AI Decision Supportcited by 2
13 more
  1. 2026AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Sciencecited by 2
  2. 2025Towards Conversational AI for Human-Machine Collaborative MLOpscited by 1
  3. 2026Alignment-Process-Outcome: Rethinking How AIs and Humans Collaboratecited by 1
  4. 2026HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualizationcited by 1
  5. 2024AI-Empowered Human Research Integrating Brain Science and Social Sciences Insightscited by 0
  6. 2025YRC-Bench: A Benchmark for Learning to Coordinate with Expertscited by 0
  7. 2025The Human-Data-Model Interaction Canvas for Visual Analyticscited by 0
  8. 2025Epitome: Pioneering an Experimental Platform for AI-Social Science Integrationcited by 0
  9. 2025Towards Human-AI Synergy in Requirements Engineering: A Framework and Preliminary Studycited by 0
  10. 2026Generativism: Toward a Learning Theory for the Age of Generative Artificial Intelligencecited by 0
  11. 2026Judgment-Grounded Expansion for Peer Review Generationcited by 0
  12. 2026AI Evaluation Should Work With Humanscited by 0
  13. 2026From Task Success to Productive Success: Evaluating Human-AI Collaboration by Quality and Costcited by 0