# Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant

arXiv: https://arxiv.org/abs/2501.17546 · HTML: https://mlsanity.com/paper/2501.17546/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 133 references that are in the map.
- First posted on arXiv 29 Jan 2025.

## Builds on (most cited first, 30 of 43)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [InterpretML: A Unified Framework for Machine Learning Interpretability](https://mlsanity.com/paper/1909.09223/) (2019)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [Human-Centered Explainable AI (XAI): From Algorithms to User Experiences](https://mlsanity.com/paper/2110.10790/) (2021)
- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making](https://mlsanity.com/paper/2102.09692/) (2021)
- [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022)
- [Interpreting Language Models with Contrastive Explanations](https://mlsanity.com/paper/2202.10419/) (2022)
- [Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach](https://mlsanity.com/paper/2002.01092/) (2020)
- ["Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans](https://mlsanity.com/paper/2001.05871/) (2020)
- [Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations](https://mlsanity.com/paper/2210.11584/) (2022)
- [Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations](https://mlsanity.com/paper/2302.02187/) (2023)
- [Contrastive Explanations for Model Interpretability](https://mlsanity.com/paper/2103.01378/) (2021)
- [explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning](https://mlsanity.com/paper/1908.00087/) (2019)
- [TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations](https://mlsanity.com/paper/2207.04154/) (2022)
- [Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision support](https://mlsanity.com/paper/2302.12389/) (2023)
- [Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanations](https://mlsanity.com/paper/1912.09024/) (2019)
- [Stakeholders in Explainable AI](https://mlsanity.com/paper/1810.00184/) (2018)
- [Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems](https://mlsanity.com/paper/2301.11333/) (2023)
- [The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems](https://mlsanity.com/paper/2008.09100/) (2020)

## Built on by (most cited first, 26 of 26)

- [Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant](https://mlsanity.com/paper/2502.01390/) (2025)
- [Deep Learning Approaches for Multimodal Intent Recognition: A Survey](https://mlsanity.com/paper/2507.22934/) (2025)
- [Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations](https://mlsanity.com/paper/2510.17256/) (2025)
- [Personalized to Persuade: The Effects of Contextualization and Warmth on Trust and Reliance in Conversational AI](https://mlsanity.com/paper/2605.31275/) (2026)
- [Exploring the Impact of Explainable AI and Cognitive Capabilities on Users' Decisions](https://mlsanity.com/paper/2505.01192/) (2025)
- [Composable Building Blocks for Controllable and Transparent Interactive AI Systems](https://mlsanity.com/paper/2506.02262/) (2025)
- [PHAX: A Structured Argumentation Framework for User-Centered Explainable AI in Public Health and Biomedical Sciences](https://mlsanity.com/paper/2507.22009/) (2025)
- [Triadic Fusion of Cognitive, Functional, and Causal Dimensions for Explainable LLMs: The TAXAL Framework](https://mlsanity.com/paper/2509.05199/) (2025)
- [Personality over Precision: Exploring the Influence of Human-Likeness on ChatGPT Use for Search](https://mlsanity.com/paper/2511.06447/) (2025)
- [AI as Teammate or Tool? A Review of Human-AI Interaction in Decision Support](https://mlsanity.com/paper/2602.15865/) (2026)
- [More Isn't Always Better: Balancing Decision Accuracy and Conformity Pressures in Multi-AI Advice](https://mlsanity.com/paper/2603.22152/) (2026)
- [What Do People Want to Know About Artificial Intelligence (AI)? The Importance of Answering End-User Questions to Explain Autonomous Vehicle (AV) Decisions](https://mlsanity.com/paper/2505.06428/) (2025)
- [The Role of Visualization in LLM-Assisted Knowledge Graph Systems: Effects on User Trust, Exploration, and Workflows](https://mlsanity.com/paper/2505.21512/) (2025)
- [Multilingual Datasets for Custom Input Extraction and Explanation Requests Parsing in Conversational XAI Systems](https://mlsanity.com/paper/2508.14982/) (2025)
- [MATCH: Engineering Transparent and Controllable Conversational XAI Systems through Composable Building Blocks](https://mlsanity.com/paper/2511.22420/) (2025)
- [Textual Explanations and Their Evaluations for Reinforcement Learning Policy](https://mlsanity.com/paper/2601.02514/) (2026)
- [The Fake Friend Dilemma: Relational Trust and the Political Economy of Conversational AI](https://mlsanity.com/paper/2601.03222/) (2026)
- [From SERPs to Sound: How Search Engine Result Pages and AI-generated Podcasts Interact to Influence User Attitudes on Controversial Topics](https://mlsanity.com/paper/2601.11282/) (2026)
- [When Life Gives You AI, Will You Turn It Into A Market for Lemons? Understanding How Information Asymmetries About AI System Capabilities Affect Market Outcomes and Adoption](https://mlsanity.com/paper/2601.21650/) (2026)
- [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026)
- [Not All Trust is the Same: Effects of Decision Workflow and Explanations in Human-AI Decision Making](https://mlsanity.com/paper/2603.05229/) (2026)
- [Persistent and Conversational Multi-Method Explainability for Trustworthy Financial AI](https://mlsanity.com/paper/2605.11687/) (2026)
- [Can Conversational XAI Improve User Performance? An Experimental Study](https://mlsanity.com/paper/2605.20439/) (2026)
- [Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness](https://mlsanity.com/paper/2605.27879/) (2026)
- [SpheriCity: Designing Trustworthy Conversational AI for Sustainability Decision Support](https://mlsanity.com/paper/2606.13854/) (2026)
- [Explainability Assistant: A Conversational XAI Interface for Interpreting Energy Consumption Models](https://mlsanity.com/paper/2609.11860/) (2026)

## Abstract (arXiv)

Explainable artificial intelligence (XAI) methods are being proposed to help interpret and understand how AI systems reach specific predictions. Inspired by prior work on conversational user interfaces, we argue that augmenting existing XAI methods with conversational user interfaces can increase user engagement and boost user understanding of the AI system. In this paper, we explored the impact of a conversational XAI interface on users' understanding of the AI system, their trust, and reliance on the AI system. In comparison to an XAI dashboard, we found that the conversational XAI interface can bring about a better understanding of the AI system among users and higher user trust. However, users of both the XAI dashboard and conversational XAI interfaces showed clear overreliance on the AI system. Enhanced conversations powered by large language model (LLM) agents amplified over-reliance. Based on our findings, we reason that the potential cause of such overreliance is the illusion of explanatory depth that is concomitant with both XAI interfaces. Our findings have important implications for designing effective conversational XAI interfaces to facilitate appropriate reliance and improve human-AI collaboration. Code can be found at https://github.com/delftcrowd/IUI2025_ConvXAI