Transcending XAI Algorithm Boundaries through End-User-Inspired Design
Not on the map. Why?
Abstract (arXiv)
The boundaries of existing explainable artificial intelligence (XAI) algorithms are confined to problems grounded in technical users' demand for explainability. This research paradigm disproportionately ignores the larger group of non-technical end users, who have a much higher demand for AI explanations in diverse explanation goals, such as making safer and better decisions and improving users' predicted outcomes. Lacking explainability-focused functional support for end users may hinder the safe and accountable use of AI in high-stakes domains, such as healthcare, criminal justice, finance, and autonomous driving systems. Built upon prior human factor analysis on end users' requirements for XAI, we identify and model four novel XAI technical problems covering the full spectrum from design to the evaluation of XAI algorithms, including edge-case-based reasoning, customizable counterfactual explanation, collapsible decision tree, and the verifiability metric to evaluate XAI utility. Based on these newly-identified research problems, we also discuss open problems in the technical development of user-centered XAI to inspire future research. Our work bridges human-centered XAI with the technical XAI community, and calls for a new research paradigm on the technical development of user-centered XAI for the responsible use of AI in critical tasks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 47 of its 105 references that are in the map. First posted on arXiv 18 Aug 2022.
Builds on
47 of 105 references.
References · 47
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
- 2017Axiomatic Attribution for Deep Networkscited by 3,384
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
- 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
- 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
- 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
- 2018Sanity Checks for Saliency Mapscited by 1,156
- 2017SmoothGrad: removing noise by adding noisecited by 1,109
35 more
- 2014Striving for Simplicity: The All Convolutional Netcited by 990
- 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
- 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
- 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 373
- 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 367
- 2019Counterfactual Visual Explanationscited by 313
- 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 297
- 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
- 2018Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negativescited by 262
- 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
- 2017Distilling a Neural Network Into a Soft Decision Treecited by 236
- 2017Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictionscited by 234
- 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 214
- 2016Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Modelscited by 213
- 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
- 2018Local Rule-Based Explanations of Black Box Decision Systemscited by 166
- 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
- 2020Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challengescited by 120
- 2019"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanationscited by 111
- 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
- 2018Interpreting CNNs via Decision Treescited by 89
- 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 88
- 2017Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciencescited by 85
- 2015Supersparse Linear Integer Models for Optimized Medical Scoring Systemscited by 76
- 2018How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanationcited by 72
- 2016Scalable Bayesian Rule Listscited by 64
- 2018RuleMatrix: Visualizing and Understanding Classifiers with Rulescited by 62
- 2014Falling Rule Listscited by 59
- 2022Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysiscited by 35
- 2017The Doctor Just Won't Accept That!cited by 21
- 2022Machine Explanations and Human Understandingcited by 20
- 2022Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?cited by 20
- 2021EUCA: the End-User-Centered Explainable AI Frameworkcited by 10
- 2022How Can We Develop Explainable Systems? Insights from a Literature Review and an Interview Studycited by 5
- 2022Can Requirements Engineering Support Explainable Artificial Intelligence? Towards a User-Centric Approach for Explainability Requirementscited by 4
Built on by
Citing papers per year: 2023: 2.
2 papers.