mlsanityData to 7 Oct 2026

The Who in XAI: How AI Background Shapes Perceptions of AI Explanations

arXiv 2107.13509 ↗First posted 28 Jul 2021cs.HC cs.AI cs.CYcited by 48

Not on the map. Why?

Abstract (arXiv)

Explainability of AI systems is critical for users to take informed actions. Understanding "who" opens the black-box of AI is just as important as opening it. We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. Quantitatively, we share user perceptions along five dimensions. Qualitatively, we describe how AI background can influence interpretations, elucidating the differences through lenses of appropriation and cognitive heuristics. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design. Carrying critical implications for the field of XAI, our findings showcase how AI generated explanations can have negative consequences despite best intentions and how that could lead to harmful manipulation of trust. We propose design interventions to mitigate them.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 40 of its 182 references that are in the map. First posted on arXiv 28 Jul 2021.

Builds on

40 of 182 references.

References · 40

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  5. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  6. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  7. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  8. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  9. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  10. 2016The Mythos of Model Interpretabilitycited by 716
  11. 2020Artificial Intelligence, Values and Alignmentcited by 443
  12. 2017Residual Attention Network for Image Classificationcited by 385
28 more
  1. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  2. 2016Image Captioning with Semantic Attentioncited by 260
  3. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 251
  4. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
  5. 2017What do we need to build explainable AI systems for the medical domain?cited by 127
  6. 2018'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisionscited by 91
  7. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  8. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  9. 2019Problem Formulation and Fairnesscited by 85
  10. 2019Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptionscited by 81
  11. 2018Boolean Decision Rules via Column Generationcited by 79
  12. 2017Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillationcited by 62
  13. 2018The age of secrecy and unfairness in recidivism predictioncited by 53
  14. 2020The Pragmatic Turn in Explainable Artificial Intelligence (XAI)cited by 53
  15. 2018Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systemscited by 47
  16. 2020Factors Influencing Perceived Fairness in Algorithmic Decision-Making: Algorithm Outcomes, Development Procedures, and Individual Differencescited by 41
  17. 2017Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanationscited by 37
  18. 2019Explainability in Human-Agent Systemscited by 34
  19. 2019Generalized Linear Rule Modelscited by 34
  20. 2021Explainable AI for Robot Failures: Generating Explanations that Improve User Assistance in Fault Recoverycited by 32
  21. 2019Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgmentcited by 31
  22. 2020Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Visioncited by 25
  23. 2020Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projectscited by 24
  24. 2020Leveraging Rationales to Improve Human Task Performancecited by 20
  25. 2017Translating Neuralesecited by 16
  26. 2018Expressing Robot Incapabilitycited by 12
  27. 2020Data Vision: Learning to See Through Algorithmic Abstractioncited by 11
  28. 2020How to Support Users in Understanding Intelligent Systems? Structuring the Discussioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 13; 2024: 6; 2025: 10; 2026: 8.

48 papers.

Sort:

By area:

  1. 2023Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanationscited by 73
  2. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  3. 2023Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAIcited by 33
  4. 2022Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theorycited by 32
  5. 2023Selective Explanations: Leveraging Human Input to Align Explainable AIcited by 29
  6. 2025Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checkingcited by 29
  7. 2022Human Interpretation of Saliency-based Explanation Over Textcited by 25
  8. 2023`It is currently hodgepodge'': Examining AI/ML Practitioners' Challenges during Co-production of Responsible AI Valuescited by 18
  9. 2023XAIR: A Framework of Explainable AI in Augmented Realitycited by 17
  10. 2022Diagnosing AI Explanation Methods with Folk Concepts of Behaviorcited by 15
  11. 2023The Co-12 Recipe for Evaluating Interpretable Part-Prototype Image Classifierscited by 15
  12. 2022The Conflict Between Explainable and Accountable Decision-Making Algorithmscited by 13
36 more
  1. 2023Explainability in AI Policies: A Critical Review of Communications, Reports, Regulations, and Standards in the EU, US, and UKcited by 13
  2. 2025DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidencecited by 10
  3. 2022Let's Go to the Alien Zoo: Introducing an Experimental Framework to Study Usability of Counterfactual Explanations for Machine Learningcited by 9
  4. 2022Mediators: Conversational Agents Explaining NLP Model Behaviorcited by 7
  5. 2022Think About the Stakeholders First! Towards an Algorithmic Transparency Playbook for Regulatory Compliancecited by 7
  6. 2022Social Construction of XAI: Do We Need One Definition to Rule Them All?cited by 6
  7. 2024The Great AI Witch Hunt: Reviewers Perception and (Mis)Conception of Generative AI in Research Writingcited by 6
  8. 2022The Risks of Machine Learning Systemscited by 5
  9. 2023Blaming Humans and Machines: What Shapes People's Reactions to Algorithmic Harmcited by 5
  10. 2023May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainabilitycited by 5
  11. 2023Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanationscited by 3
  12. 2023Exploring Practitioner Perspectives On Training Data Attribution Explanationscited by 3
  13. 2024Towards User-Focused Research in Training Data Attribution for Human-Centered Explainable AIcited by 3
  14. 2025Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviourcited by 3
  15. 2025XAgen: An Explainability Tool for Identifying and Correcting Failures in Multi-Agent Workflowscited by 3
  16. 2024People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AIcited by 2
  17. 2024XEQ Scale for Evaluating XAI Experience Qualitycited by 2
  18. 2022The Influence of Explainable Artificial Intelligence: Nudging Behaviour or Boosting Capability?cited by 1
  19. 2023Towards Feminist Intersectional XAI: From Explainability to Response-Abilitycited by 1
  20. 2023Unjustified Sample Sizes and Generalizations in Explainable AI Research: Principles for More Inclusive User Studiescited by 1
  21. 2024Scenarios and Approaches for Situated Natural Language Explanationscited by 1
  22. 2025"Impressively Scary:" Exploring User Perceptions and Reactions to Unraveling Machine Learning Models in Social Media Applicationscited by 1
  23. 2025Predicting Satisfaction of Counterfactual Explanations from Human Ratings of Explanatory Qualitiescited by 1
  24. 2026Comparables XAI: Faithful Example-based AI Explanations with Counterfactual Trace Adjustmentscited by 1
  25. 2022Tell Me Something That Will Help Me Trust You: A Survey of Trust Calibration in Human-Agent Interactioncited by 0
  26. 2024Exploring the Effect of Explanation Content and Format on User Comprehension and Trust in Healthcarecited by 0
  27. 2025What Do People Want to Know About Artificial Intelligence (AI)? The Importance of Answering End-User Questions to Explain Autonomous Vehicle (AV) Decisionscited by 0
  28. 2025Do Ethical AI Principles Matter to Users? A Large-Scale Analysis of User Sentiment and Satisfactioncited by 0
  29. 2025Beyond Technocratic XAI: The Who, What & How in Explanation Designcited by 0
  30. 2026Emergent, not Immanent: A Baradian Reading of Explainable AIcited by 0
  31. 2026Power Echoes: Investigating Moderation Biases in Online Power-Asymmetric Conflictscited by 0
  32. 2026Ceci n'est pas une explication: Evaluating Explanation Failures as Explainability Pitfalls in Language Learning Systemscited by 0
  33. 2026Persistent and Conversational Multi-Method Explainability for Trustworthy Financial AIcited by 0
  34. 2026SketchXplain: Intuitive Visual Explanations of Image Classifiers with Sketchescited by 0
  35. 2026Position: We Need Practical AI Alignment Methods to Mirror Human Reasoningcited by 0
  36. 2026XAI-Arena: Can LLMs Assess the Quality of XAI Explanations?cited by 0