# Measuring User Understanding in Dialogue-based XAI Systems

arXiv: https://arxiv.org/abs/2408.06960 · HTML: https://mlsanity.com/paper/2408.06960/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 13 Aug 2024.

## Builds on (most cited first, 15 of 15)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020)
- [TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations](https://mlsanity.com/paper/2207.04154/) (2022)
- [One Explanation Does Not Fit All: The Promise of Interactive Explanations for Machine Learning Transparency](https://mlsanity.com/paper/2001.09734/) (2020)
- [The Grammar of Interactive Explanatory Model Analysis](https://mlsanity.com/paper/2005.00497/) (2020)
- [The Impact of Imperfect XAI on Human-AI Decision-Making](https://mlsanity.com/paper/2307.13566/) (2023)
- [What Would You Ask the Machine Learning Model? Identification of User Needs for Model Explanations Based on Human-Model Conversations](https://mlsanity.com/paper/2002.05674/) (2020)
- [InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanations](https://mlsanity.com/paper/2310.05592/) (2023)
- [Forms of Understanding for XAI-Explanations](https://mlsanity.com/paper/2311.08760/) (2023)

## Built on by (most cited first, 6 of 6)

- [Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant](https://mlsanity.com/paper/2501.17546/) (2025)
- [Investigating Co-Constructive Behavior of Large Language Models in Explanation Dialogues](https://mlsanity.com/paper/2504.18483/) (2025)
- [PHAX: A Structured Argumentation Framework for User-Centered Explainable AI in Public Health and Biomedical Sciences](https://mlsanity.com/paper/2507.22009/) (2025)
- [Multilingual Datasets for Custom Input Extraction and Explanation Requests Parsing in Conversational XAI Systems](https://mlsanity.com/paper/2508.14982/) (2025)
- [Can Conversational XAI Improve User Performance? An Experimental Study](https://mlsanity.com/paper/2605.20439/) (2026)
- [Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness](https://mlsanity.com/paper/2605.27879/) (2026)

## Abstract (arXiv)

The field of eXplainable Artificial Intelligence (XAI) is increasingly recognizing the need to personalize and/or interactively adapt the explanation to better reflect users' explanation needs. While dialogue-based approaches to XAI have been proposed recently, the state-of-the-art in XAI is still characterized by what we call one-shot, non-personalized and one-way explanations. In contrast, dialogue-based systems that can adapt explanations through interaction with a user promise to be superior to GUI-based or dashboard explanations as they offer a more intuitive way of requesting information. In general, while interactive XAI systems are often evaluated in terms of user satisfaction, there are limited studies that access user's objective model understanding. This is in particular the case for dialogue-based XAI approaches. In this paper, we close this gap by carrying out controlled experiments within a dialogue framework in which we measure understanding of users in three phases by asking them to simulate the predictions of the model they are learning about. By this, we can quantify the level of (improved) understanding w.r.t. how the model works, comparing the state prior, and after the interaction. We further analyze the data to reveal patterns of how the interaction between groups with high vs. low understanding gain differ. Overall, our work thus contributes to our understanding about the effectiveness of XAI approaches.