mlsanityData to 7 Oct 2026

Question-Driven Design Process for Explainable AI User Experiences

arXiv 2104.03483 ↗First posted 8 Apr 2021cs.HC cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

A pervasive design issue of AI systems is their explainability--how to provide appropriate information to help users understand the AI. The technical field of explainable AI (XAI) has produced a rich toolbox of techniques. Designers are now tasked with the challenges of how to select the most suitable XAI techniques and translate them into UX solutions. Informed by our previous work studying design challenges around XAI UX, this work proposes a design process to tackle these challenges. We review our and related prior work to identify requirements that the process should fulfill, and accordingly, propose a Question-Driven Design Process that grounds the user needs, choices of XAI techniques, design, and evaluation of XAI UX all in the user questions. We provide a mapping guide between prototypical user questions and exemplars of XAI techniques to reframe the technical space of XAI, also serving as boundary objects to support collaboration between designers and AI engineers. We demonstrate it with a use case of designing XAI for healthcare adverse events prediction, and discuss lessons learned for tackling design challenges of AI systems.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 19 of its 56 references that are in the map. First posted on arXiv 8 Apr 2021.

Builds on

19 of 56 references.

References · 19

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  6. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  7. 2018AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Biascited by 319
  8. 2018Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negativescited by 261
  9. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 251
  10. 2019Explainable Machine Learning in Deploymentcited by 210
  11. 2019One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniquescited by 164
  12. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
7 more
  1. 2018Explaining Explanations in AIcited by 136
  2. 2020Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertaintycited by 128
  3. 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86
  4. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  5. 2018Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systemscited by 47
  6. 2020CheXplain: Enabling Physicians to Explore and UnderstandData-Driven, AI-Enabled Medical Imaging Analysiscited by 40
  7. 2019Explainability in Human-Agent Systemscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 6; 2024: 6; 2025: 7; 2026: 2.

31 papers.

Sort:

By area:

  1. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  2. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 84
  3. 2022Investigating Explainability of Generative AI for Code through Scenario-based Designcited by 47
  4. 2023Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAIcited by 33
  5. 2022Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and Engineeringcited by 20
  6. 2023Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experiencecited by 18
  7. 2023"It Felt Like Having a Second Mind": Investigating Human-AI Co-creativity in Prewriting with Large Language Modelscited by 15
  8. 2022Towards a multi-stakeholder value-based assessment framework for algorithmic systemscited by 12
  9. 2022Explainable AI for Bioinformatics: Methods, Tools, and Applicationscited by 11
  10. 2022Towards a Roadmap on Software Engineering for Responsible AIcited by 9
  11. 2022A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunitiescited by 8
  12. 2022Mediators: Conversational Agents Explaining NLP Model Behaviorcited by 7
19 more
  1. 2024CoXQL: A Dataset for Parsing Explanation Requests in Conversational XAI Systemscited by 5
  2. 2021Explanation Strategies as an Empirical-Analytical Lens for Socio-Technical Contextualization of Machine Learning Interpretabilitycited by 4
  3. 2025Explainable AI-Based Interface System for Weather Forecasting Modelcited by 4
  4. 2023Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AIcited by 3
  5. 2024Information That Matters: Exploring Information Needs of People Affected by Algorithmic Decisionscited by 3
  6. 2024Responsible AI Question Bank: A Comprehensive Tool for AI Risk Assessmentcited by 2
  7. 2023I-CEE: Tailoring Explanations of Image Classification Models to User Expertisecited by 1
  8. 2024Aligning XAI with EU Regulations for Smart Biomedical Devices: A Methodology for Compliance Analysiscited by 1
  9. 2024Output Scouting: Auditing Large Language Models for Catastrophic Responsescited by 1
  10. 2025ArchiveGPT: A human-centered evaluation of using a vision language model for image cataloguingcited by 1
  11. 2025Holistic Explainable AI (H-XAI): Extending Transparency Beyond Developers in AI-Driven Decision Makingcited by 1
  12. 2025PILAR: Personalizing Augmented Reality Interactions with LLM-based Human-Centric and Trustworthy Explanations for Daily Use Casescited by 1
  13. 2026Assessing Model-Agnostic XAI Methods against EU AI Act Explainability Requirementscited by 1
  14. 2023Towards Real Smart Apps: Investigating Human-AI Interactions in Smartphone On-Device AI Appscited by 0
  15. 2024Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language Modelscited by 0
  16. 2025E-LENS: User Requirements-Oriented AI Ethics Assurancecited by 0
  17. 2025A Question Bank to Assess AI Inclusivity: Mapping out the Journey from Diversity Errors to Inclusion Excellencecited by 0
  18. 2025DeformTune: A Deformable XAI Music Prototype for Non-Musicianscited by 0
  19. 2026Scientific Theory of a Black-Box: A Life Cycle-Scale XAI Framework Based on Constructive Empiricismcited by 0