mlsanityData to 7 Oct 2026

Invisible Users: Uncovering End-Users' Requirements for Explainable AI via Explanation Forms and Goals

arXiv 2302.06609 ↗First posted 10 Feb 2023cs.HC cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Non-technical end-users are silent and invisible users of the state-of-the-art explainable artificial intelligence (XAI) technologies. Their demands and requirements for AI explainability are not incorporated into the design and evaluation of XAI techniques, which are developed to explain the rationales of AI decisions to end-users and assist their critical decisions. This makes XAI techniques ineffective or even harmful in high-stakes applications, such as healthcare, criminal justice, finance, and autonomous driving systems. To systematically understand end-users' requirements to support the technical development of XAI, we conducted the EUCA user study with 32 layperson participants in four AI-assisted critical tasks. The study identified comprehensive user requirements for feature-, example-, and rule-based XAI techniques (manifested by the end-user-friendly explanation forms) and XAI evaluation objectives (manifested by the explanation goals), which were shown to be helpful to directly inspire the proposal of new XAI algorithms and evaluation metrics. The EUCA study findings, the identified explanation forms and goals for technical specification, and the EUCA study dataset support the design and evaluation of end-user-centered XAI techniques for accessible, safe, and accountable AI.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 44 of its 82 references that are in the map. First posted on arXiv 10 Feb 2023.

Builds on

44 of 82 references.

References · 44

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  4. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  5. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  6. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  7. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  8. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  9. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  10. 2014Understanding Deep Image Representations by Inverting Themcited by 399
  11. 2019Counterfactual Visual Explanationscited by 312
  12. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
32 more
  1. 2018Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negativescited by 261
  2. 2017Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictionscited by 234
  3. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  4. 2016Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Modelscited by 212
  5. 2019Explainable Machine Learning in Deploymentcited by 210
  6. 2018Local Rule-Based Explanations of Black Box Decision Systemscited by 165
  7. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  8. 2019"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanationscited by 111
  9. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  10. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  11. 2018Interpreting CNNs via Decision Treescited by 89
  12. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 87
  13. 2017Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciencescited by 84
  14. 2015Supersparse Linear Integer Models for Optimized Medical Scoring Systemscited by 76
  15. 2020Human Factors in Model Interpretability: Industry Practices, Challenges, and Needscited by 73
  16. 2018How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanationcited by 71
  17. 2016Scalable Bayesian Rule Listscited by 64
  18. 2021Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needscited by 64
  19. 2018RuleMatrix: Visualizing and Understanding Classifiers with Rulescited by 62
  20. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  21. 2020A systematic review and taxonomy of explanations in decision support and recommender systemscited by 45
  22. 2020How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labelscited by 39
  23. 2021Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabicited by 37
  24. 2022Visual correspondence-based explanations improve AI robustness and human-AI team accuracycited by 37
  25. 2022Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysiscited by 34
  26. 2022Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?cited by 19
  27. 2017Towards the Augmented Pathologist: Challenges of Explainable-AI in Digital Pathologycited by 13
  28. 2019Artificial Intelligence in Glioma Imaging: Challenges and Advancescited by 10
  29. 2022How Can We Develop Explainable Systems? Insights from a Literature Review and an Interview Studycited by 5
  30. 2022Can Requirements Engineering Support Explainable Artificial Intelligence? Towards a User-Centric Approach for Explainability Requirementscited by 4
  31. 2019Personalized explanation in machine learning: A conceptualizationcited by 2
  32. 2022Transcending XAI Algorithm Boundaries through End-User-Inspired Designcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024AI Readiness in Healthcare through Storytelling XAIcited by 4
  2. 2024Towards User-Focused Research in Training Data Attribution for Human-Centered Explainable AIcited by 3
  3. 2024Explainable Interface for Human-Autonomy Teaming: A Surveycited by 2
  4. 2026Evaluating RE Practices for Explainability: Synthesizing Insights from Daimler Truck into an Explainable RE Framework Proposalcited by 0