mlsanityData to 14 Oct 2026

Questioning the AI: Informing Design Practices for Explainable AI User Experiences

arXiv 2001.02478 ↗First posted 8 Jan 2020cs.HC cs.AI cs.LGcited by 253

Not on the map. Why?

Abstract (arXiv)

A surge of interest in explainable AI (XAI) has led to a vast collection of algorithmic work on the topic. While many recognize the necessity to incorporate explainability features in AI systems, how to address real-world user needs for understanding AI remains an open question. By interviewing 20 UX and design practitioners working on various AI products, we seek to identify gaps between the current XAI algorithmic work and practices to create explainable AI products. To do so, we develop an algorithm-informed XAI question bank in which user needs for explainability are represented as prototypical questions users might ask about the AI, and use it as a study probe. Our work contributes insights into the design space of XAI, informs efforts to support design practices in this space, and identifies opportunities for future XAI work. We also provide an extended XAI question bank and discuss how it can be used for creating user-centered XAI.

Key facts. Cited by 253 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 28; 2022: 41; 2023: 49; 2024: 49; 2025: 45; 2026: 41.

253 papers.

Sort:

By area:

  1. 2021Expanding Explainability: Towards Social Transparency in AI systemscited by 124
  2. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  3. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 123
  4. 2023The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practicecited by 87
  5. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 86
  6. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  7. 2022Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderationcited by 67
  8. 2022TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversationscited by 66
  9. 2021Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needscited by 64
  10. 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 64
  11. 2022"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interactioncited by 63
  12. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 62
38 more
  1. 2024Design Principles for Generative AI Applicationscited by 57
  2. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gapcited by 56
  3. 2022Rethinking Explainability as a Dialogue: A Practitioner's Perspectivecited by 52
  4. 2021The Who in XAI: How AI Background Shapes Perceptions of AI Explanationscited by 48
  5. 2022Investigating Explainability of Generative AI for Code through Scenario-based Designcited by 47
  6. 2023"HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social mediacited by 45
  7. 2022Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AIcited by 40
  8. 2024Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Makingcited by 38
  9. 2021Explainability Pitfalls: Beyond Dark Patterns in Explainable AIcited by 36
  10. 2023Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAIcited by 33
  11. 2022Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theorycited by 32
  12. 2021Question-Driven Design Process for Explainable AI User Experiencescited by 31
  13. 2023ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writingcited by 31
  14. 2024On the Challenges and Opportunities in Generative AIcited by 31
  15. 2021Explainable Medical Imaging AI Needs Human-Centered Design: Guidelines and Evidence from a Systematic Reviewcited by 30
  16. 2023Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosiscited by 30
  17. 2023Selective Explanations: Leveraging Human Input to Align Explainable AIcited by 29
  18. 2024Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decompositioncited by 29
  19. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 29
  20. 2021Towards Relatable Explainable AI with the Perceptual Processcited by 27
  21. 2022From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible MLcited by 27
  22. 2025Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistantcited by 26
  23. 2022Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questionscited by 23
  24. 2021Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loopcited by 21
  25. 2022ProtoTEx: Explaining Model Decisions with Prototype Tensorscited by 21
  26. 2023Toward General Design Principles for Generative AI Applicationscited by 21
  27. 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
  28. 2021How Much Automation Does a Data Scientist Want?cited by 20
  29. 2022Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and Engineeringcited by 20
  30. 2022Seamful XAI: Operationalizing Seamful Design in Explainable AIcited by 20
  31. 2022Explainable Artificial Intelligence (XAI) from a user perspective- A synthesis of prior literature and problematizing avenues for future researchcited by 20
  32. 2023Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experiencecited by 18
  33. 2023Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorationscited by 18
  34. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  35. 2023XAIR: A Framework of Explainable AI in Augmented Realitycited by 17
  36. 2023"It Felt Like Having a Second Mind": Investigating Human-AI Co-creativity in Prewriting with Large Language Modelscited by 17
  37. 2021An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainabilitycited by 16
  38. 2022Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Workcited by 15

50 most cited of 253 All 253 (JSON)