mlsanityData to 7 Oct 2026

"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction

arXiv 2210.03735 ↗First posted 2 Oct 2022cs.HC cs.AI cs.CVcited by 63

Not on the map. Why?

Abstract (arXiv)

Despite the proliferation of explainable AI (XAI) methods, little is understood about end-users' explainability needs and behaviors around XAI explanations. To address this gap and contribute to understanding how explainability can support human-AI interaction, we conducted a mixed-methods study with 20 end-users of a real-world AI application, the Merlin bird identification app, and inquired about their XAI needs, uses, and perceptions. We found that participants desire practically useful information that can improve their collaboration with the AI, more so than technical system details. Relatedly, participants intended to use XAI explanations for various purposes beyond understanding the AI's outputs: calibrating trust, improving their task skills, changing their behavior to supply better inputs to the AI, and giving constructive feedback to developers. Finally, among existing XAI approaches, participants preferred part-based explanations that resemble human reasoning and explanations. We discuss the implications of our findings and provide recommendations for future XAI design.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 82 of its 160 references that are in the map. First posted on arXiv 2 Oct 2022.

Builds on

82 of 160 references.

References · 82

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  4. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  8. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  9. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  10. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  11. 2018Sanity Checks for Saliency Mapscited by 1,149
  12. 2020Concept Bottleneck Modelscited by 1,022
38 more
  1. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  2. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  3. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  4. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  5. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  6. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  7. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  8. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  9. 2019Counterfactual Visual Explanationscited by 312
  10. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  11. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  12. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 257
  13. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  14. 2020Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Surveycited by 252
  15. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 251
  16. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  17. 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 213
  18. 2019Explainable Machine Learning in Deploymentcited by 210
  19. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  20. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 163
  21. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  22. 2019Seeing What a GAN Cannot Generatecited by 148
  23. 2018Explaining Explanations in AIcited by 136
  24. 2019What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Usecited by 134
  25. 2018Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networkscited by 128
  26. 2021Expanding Explainability: Towards Social Transparency in AI systemscited by 124
  27. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  28. 2021Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypescited by 120
  29. 2021What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Researchcited by 119
  30. 2021Do Concept Bottleneck Models Learn as Intended?cited by 114
  31. 2020Explainable deep learning models in medical image analysiscited by 111
  32. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  33. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  34. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  35. 2020The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategiescited by 91
  36. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  37. 2022B-cos Networks: Alignment is All We Need for Interpretabilitycited by 88
  38. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 87

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 22; 2025: 18; 2026: 9.

63 papers.

Sort:

By area:

  1. 2023Concept-based Explainable Artificial Intelligence: A Surveycited by 97
  2. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  3. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  4. 2024Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decompositioncited by 29
  5. 2023The Impact of Imperfect XAI on Human-AI Decision-Makingcited by 21
  6. 2023Humans, AI, and Context: Understanding End-Users' Trust in a Real-World Computer Vision Applicationcited by 14
  7. 2024EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurationscited by 14
  8. 2024Explainable Predictive Maintenance: A Survey of Current Methods, Challenges and Opportunitiescited by 10
  9. 2023Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Mediacited by 9
  10. 2024Incremental XAI: Memorable Understanding of AI with Incremental Explanationscited by 8
  11. 2024Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaborationcited by 8
  12. 2025Is Trust Correlated With Explainability in AI? A Meta-Analysiscited by 7
38 more
  1. 2025Improving Human Verification of LLM Reasoning through Interactive Explanation Interfacescited by 7
  2. 2023Selective Concept Models: Permitting Stakeholder Customisation at Test-Timecited by 6
  3. 2023May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainabilitycited by 5
  4. 2024Interrogating AI: Characterizing Emergent Playful Interactions with ChatGPTcited by 5
  5. 2024Allowing humans to interactively guide machines where to look does not always improve human-AI team's classification accuracycited by 5
  6. 2024How Mature is Requirements Engineering for AI-based Systems? A Systematic Mapping Study on Practices, Challenges, and Future Research Directionscited by 5
  7. 2025A Framework for Responsible AI Systems: Building Societal Trust through Domain Definition, Trustworthy AI Design, Auditability, Accountability, and Governancecited by 5
  8. 2025Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Softwarecited by 5
  9. 2025Interaction as Intelligence: Deep Research With Human-AI Partnershipcited by 5
  10. 2025GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Designcited by 5
  11. 2024Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learningcited by 4
  12. 2023Towards Self-Explainability of Deep Neural Networks with Heatmap Captioning and Large-Language Modelscited by 3
  13. 2023An Information Bottleneck Characterization of the Understanding-Workload Tradeoffcited by 3
  14. 2024Information That Matters: Exploring Information Needs of People Affected by Algorithmic Decisionscited by 3
  15. 2024Tell me more: Intent Fulfilment Framework for Enhancing User Experiences in Conversational XAIcited by 3
  16. 2024Semantic Prototypes: Enhancing Transparency Without Black Boxescited by 3
  17. 2024Bridging Human Concepts and Computer Vision for Explainable Face Verificationcited by 2
  18. 2024People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AIcited by 2
  19. 2024CoProNN: Concept-based Prototypical Nearest Neighbors for Explaining Vision Modelscited by 2
  20. 2024Explainable Interface for Human-Autonomy Teaming: A Surveycited by 2
  21. 2024XEQ Scale for Evaluating XAI Experience Qualitycited by 2
  22. 2025A Survey on Interpretability in Visual Recognitioncited by 2
  23. 2026Mixture of Concept Bottleneck Expertscited by 2
  24. 2026Position: Explainability Research Must Prioritize Foundations over Ad-hoc Methodscited by 2
  25. 2023UFO: A unified method for controlling Understandability and Faithfulness Objectives in concept-based explanations for CNNscited by 1
  26. 2023Black-box Attacks on Image Activity Prediction and its Natural Language Explanationscited by 1
  27. 2023Representing visual classification as a linear combination of wordscited by 1
  28. 2023Estimation of Concept Explanations Should be Uncertainty Awarecited by 1
  29. 2024Contrastive Pretraining for Visual Concept Explanations of Socioeconomic Outcomescited by 1
  30. 2025"Impressively Scary:" Exploring User Perceptions and Reactions to Unraveling Machine Learning Models in Social Media Applicationscited by 1
  31. 2025Conversational Explanations: Discussing Explainable AI with Non-AI Expertscited by 1
  32. 2025Interactivity x Explainability: Toward Understanding How Interactivity Can Improve Computer Vision Explanationscited by 1
  33. 2025X-SiT: Inherently Interpretable Surface Vision Transformers for Dementia Diagnosiscited by 1
  34. 2025Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behaviorcited by 1
  35. 2026Designing KRIYA: An AI Companion for Wellbeing Self-Reflectioncited by 1
  36. 2026Comparables XAI: Faithful Example-based AI Explanations with Counterfactual Trace Adjustmentscited by 1
  37. 2023More efficient manual review of automatically transcribed tabular datacited by 0
  38. 2024Interaction as Explanation: A User Interaction-based Method for Explaining Image Classification Modelscited by 0

50 most cited of 63 All 63 (JSON)