mlsanityData to 14 Oct 2026

Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant

arXiv 2501.17546 ↗First posted 29 Jan 2025cs.HC cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Explainable artificial intelligence (XAI) methods are being proposed to help interpret and understand how AI systems reach specific predictions. Inspired by prior work on conversational user interfaces, we argue that augmenting existing XAI methods with conversational user interfaces can increase user engagement and boost user understanding of the AI system. In this paper, we explored the impact of a conversational XAI interface on users' understanding of the AI system, their trust, and reliance on the AI system. In comparison to an XAI dashboard, we found that the conversational XAI interface can bring about a better understanding of the AI system among users and higher user trust. However, users of both the XAI dashboard and conversational XAI interfaces showed clear overreliance on the AI system. Enhanced conversations powered by large language model (LLM) agents amplified over-reliance. Based on our findings, we reason that the potential cause of such overreliance is the illusion of explanatory depth that is concomitant with both XAI interfaces. Our findings have important implications for designing effective conversational XAI interfaces to facilitate appropriate reliance and improve human-AI collaboration. Code can be found at https://github.com/delftcrowd/IUI2025_ConvXAI

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 43 of its 133 references that are in the map. First posted on arXiv 29 Jan 2025.

Builds on

43 of 133 references.

References · 43

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
  5. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  6. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
  7. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 302
  8. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  9. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
  10. 2019InterpretML: A Unified Framework for Machine Learning Interpretabilitycited by 246
  11. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 222
  12. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
31 more
  1. 2020A Survey on Conversational Recommender Systemscited by 143
  2. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  3. 2021What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Researchcited by 119
  4. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  5. 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
  6. 2022Interpreting Language Models with Contrastive Explanationscited by 92
  7. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  8. 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86
  9. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 85
  10. 2023Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanationscited by 83
  11. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  12. 2019explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learningcited by 70
  13. 2022TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversationscited by 66
  14. 2023Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision supportcited by 66
  15. 2019Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanationscited by 64
  16. 2018Stakeholders in Explainable AIcited by 48
  17. 2023Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systemscited by 41
  18. 2020The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systemscited by 34
  19. 2021CRSLab: An Open-Source Toolkit for Building Conversational Recommender Systemcited by 34
  20. 2023ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writingcited by 31
  21. 2022Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Makingcited by 28
  22. 2020Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government Contextscited by 22
  23. 2022Seamful XAI: Operationalizing Seamful Design in Explainable AIcited by 20
  24. 2023Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorationscited by 18
  25. 2022Diagnosing AI Explanation Methods with Folk Concepts of Behaviorcited by 15
  26. 2024Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Makingcited by 14
  27. 2021Roadmap of Designing Cognitive Metrics for Explainable Artificial Intelligence (XAI)cited by 13
  28. 2022MACE: An Efficient Model-Agnostic Framework for Counterfactual Explanationcited by 12
  29. 2023On the Impact of Explanations on Understanding of Algorithmic Decision-Makingcited by 9
  30. 2024To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systemscited by 7
  31. 2024Measuring User Understanding in Dialogue-based XAI Systemscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 14.

26 papers.

Sort:

By area:

  1. 2025Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistantcited by 54
  2. 2025Deep Learning Approaches for Multimodal Intent Recognition: A Surveycited by 6
  3. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanationscited by 2
  4. 2026Personalized to Persuade: The Effects of Contextualization and Warmth on Trust and Reliance in Conversational AIcited by 2
  5. 2025Exploring the Impact of Explainable AI and Cognitive Capabilities on Users' Decisionscited by 1
  6. 2025Composable Building Blocks for Controllable and Transparent Interactive AI Systemscited by 1
  7. 2025PHAX: A Structured Argumentation Framework for User-Centered Explainable AI in Public Health and Biomedical Sciencescited by 1
  8. 2025Triadic Fusion of Cognitive, Functional, and Causal Dimensions for Explainable LLMs: The TAXAL Frameworkcited by 1
  9. 2025Personality over Precision: Exploring the Influence of Human-Likeness on ChatGPT Use for Searchcited by 1
  10. 2026AI as Teammate or Tool? A Review of Human-AI Interaction in Decision Supportcited by 1
  11. 2026More Isn't Always Better: Balancing Decision Accuracy and Conformity Pressures in Multi-AI Advicecited by 1
  12. 2025What Do People Want to Know About Artificial Intelligence (AI)? The Importance of Answering End-User Questions to Explain Autonomous Vehicle (AV) Decisionscited by 0
14 more
  1. 2025The Role of Visualization in LLM-Assisted Knowledge Graph Systems: Effects on User Trust, Exploration, and Workflowscited by 0
  2. 2025Multilingual Datasets for Custom Input Extraction and Explanation Requests Parsing in Conversational XAI Systemscited by 0
  3. 2025MATCH: Engineering Transparent and Controllable Conversational XAI Systems through Composable Building Blockscited by 0
  4. 2026Textual Explanations and Their Evaluations for Reinforcement Learning Policycited by 0
  5. 2026The Fake Friend Dilemma: Relational Trust and the Political Economy of Conversational AIcited by 0
  6. 2026From SERPs to Sound: How Search Engine Result Pages and AI-generated Podcasts Interact to Influence User Attitudes on Controversial Topicscited by 0
  7. 2026When Life Gives You AI, Will You Turn It Into A Market for Lemons? Understanding How Information Asymmetries About AI System Capabilities Affect Market Outcomes and Adoptioncited by 0
  8. 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Makingcited by 0
  9. 2026Not All Trust is the Same: Effects of Decision Workflow and Explanations in Human-AI Decision Makingcited by 0
  10. 2026Persistent and Conversational Multi-Method Explainability for Trustworthy Financial AIcited by 0
  11. 2026Can Conversational XAI Improve User Performance? An Experimental Studycited by 0
  12. 2026Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulnesscited by 0
  13. 2026SpheriCity: Designing Trustworthy Conversational AI for Sustainability Decision Supportcited by 0
  14. 2026Explainability Assistant: A Conversational XAI Interface for Interpreting Energy Consumption Modelscited by 0