mlsanityData to 7 Oct 2026

Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies

arXiv 2112.11471 ↗First posted 21 Dec 2021cs.AI cs.CL cs.CYcited by 85

Not on the map. Why?

Abstract (arXiv)

As AI systems demonstrate increasingly strong predictive performance, their adoption has grown in numerous domains. However, in high-stakes domains such as criminal justice and healthcare, full automation is often not desirable due to safety, ethical, and legal concerns, yet fully manual approaches can be inaccurate and time consuming. As a result, there is growing interest in the research community to augment human decision making with AI assistance. Besides developing AI technologies for this purpose, the emerging field of human-AI decision making must embrace empirical approaches to form a foundational understanding of how humans interact and work with AI to make decisions. To invite and help structure research efforts towards a science of understanding and improving human-AI decision making, we survey recent literature of empirical human-subject studies on this topic. We summarize the study design choices made in over 100 papers in three important aspects: (1) decision tasks, (2) AI models and AI assistance elements, and (3) evaluation metrics. For each aspect, we summarize current trends, discuss gaps in current practices of the field, and make a list of recommendations for future research. Our survey highlights the need to develop common frameworks to account for the design and research spaces of human-AI decision making, so that researchers can make rigorous choices in study design, and the research community can build on each other's work and produce generalizable scientific knowledge. We also hope this survey will serve as a bridge for HCI and AI communities to work together to mutually shape the empirical science and computational technologies for human-AI decision making.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 65 of its 160 references that are in the map. First posted on arXiv 21 Dec 2021.

Builds on

65 of 160 references.

References · 65

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  8. 2018Sanity Checks for Saliency Mapscited by 1,149
  9. 2018Model Cards for Model Reportingcited by 1,068
  10. 2019Attention is not Explanationcited by 916
  11. 2016The Mythos of Model Interpretabilitycited by 716
  12. 2019Attention is not not Explanationcited by 629
38 more
  1. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  2. 2019Is Attention Interpretable?cited by 387
  3. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 364
  4. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  5. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  6. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  7. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 251
  8. 2017Advances in Variational Inferencecited by 236
  9. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  10. 2018Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learningcited by 203
  11. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  12. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  13. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  14. 2021The Values Encoded in Machine Learning Researchcited by 172
  15. 2020Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AIcited by 166
  16. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  17. 2020Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scanscited by 158
  18. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
  19. 2018Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontierscited by 117
  20. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  21. 2019When Explanations Lie: Why Many Modified BP Attributions Failcited by 92
  22. 2018'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisionscited by 91
  23. 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86
  24. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  25. 2020A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scorescited by 74
  26. 2018How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanationcited by 71
  27. 2019An Evaluation of the Human-Interpretability of Explanationcited by 71
  28. 2018Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Makingcited by 66
  29. 2021Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needscited by 64
  30. 2021Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Makingcited by 58
  31. 2018Bayesian Layers: A Module for Neural Network Uncertaintycited by 57
  32. 2021Explanation-Based Human Debugging of NLP Models: A Surveycited by 53
  33. 2018What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Playcited by 52
  34. 2018Do Explanations make VQA Models more Predictable to a Human?cited by 51
  35. 2021The Flaws of Policies Requiring Human Oversight of Government Algorithmscited by 44
  36. 2013Correlation and variable importance in random forestscited by 41
  37. 2019Why Does My Model Fail? Contrastive Local Explanations for Retail Forecastingcited by 31
  38. 2021Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiativecited by 28

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 23; 2024: 25; 2025: 14; 2026: 13.

85 papers.

Sort:

By area:

  1. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  2. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 84
  3. 2023Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanationscited by 80
  4. 2023Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanationscited by 73
  5. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gapcited by 56
  6. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51
  7. 2022Designing for Responsible Trust in AI Systems: A Communication Perspectivecited by 48
  8. 2024Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Directioncited by 46
  9. 2023Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systemscited by 40
  10. 2024Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Makingcited by 38
  11. 2022Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Supportcited by 37
  12. 2023Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosiscited by 30
38 more
  1. 2022A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Makingcited by 29
  2. 2023Selective Explanations: Leveraging Human Input to Align Explainable AIcited by 29
  3. 2022Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Makingcited by 28
  4. 2023Human-Aligned Calibration for AI-Assisted Decision Makingcited by 28
  5. 2023Designing Decision Support Systems Using Counterfactual Prediction Setscited by 26
  6. 2023Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Makingcited by 25
  7. 2024Towards Human-AI Complementarity with Prediction Setscited by 25
  8. 2024Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Functioncited by 24
  9. 2023Evaluating General-Purpose AI with Psychometricscited by 23
  10. 2022A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementaritycited by 22
  11. 2022Machine Explanations and Human Understandingcited by 20
  12. 2023Effective Human-AI Teams via Learned Natural Language Rules and Onboardingcited by 20
  13. 2023Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Makingcited by 18
  14. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  15. 2024Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Makingcited by 14
  16. 2023Learning Personalized Decision Support Policiescited by 13
  17. 2024Modulating Language Model Experiences through Frictionscited by 13
  18. 2022Algorithmic Assistance with Recommendation-Dependent Preferencescited by 11
  19. 2023Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Makingcited by 11
  20. 2023Using AI Uncertainty Quantification to Improve Human Decision-Makingcited by 11
  21. 2024Characterizing and modeling harms from interactions with design patterns in AI interfacescited by 11
  22. 2025AI Mismatches: Identifying Potential Algorithmic Harms Before AI Developmentcited by 10
  23. 2023Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AIcited by 9
  24. 2024Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Setscited by 9
  25. 2025From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysiscited by 9
  26. 2025Generative AI in Knowledge Work: Design Implications for Data Navigation and Decision-Makingcited by 9
  27. 2023Capturing Humans' Mental Models of AI: An Item Response Theory Approachcited by 8
  28. 2024To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systemscited by 7
  29. 2023Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)cited by 6
  30. 2023Training Towards Critical Use: Learning to Situate AI Predictions Relative to Human Knowledgecited by 6
  31. 2024Understanding the Effects of Miscalibrated AI Confidence on User Trust, Reliance, and Decision Efficacycited by 6
  32. 2025Human-AI Collaborative Uncertainty Quantificationcited by 6
  33. 2023Collaborative Learning with Artificial Intelligence Speakers (CLAIS): Pre-Service Elementary Science Teachers' Responses to the Prototypecited by 5
  34. 2024Beyond Recommender: An Exploratory Study of the Effects of Different AI Roles in AI-Assisted Decision Makingcited by 5
  35. 2024An Empirical Exploration of Trust Dynamics in LLM Supply Chainscited by 5
  36. 2025Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translationcited by 5
  37. 2023Learning Human-Compatible Representations for Case-Based Decision Supportcited by 4
  38. 2023Thinking Assistants: LLM-Based Conversational Assistants that Help Users Think By Asking rather than Answeringcited by 4

50 most cited of 85 All 85 (JSON)