# Show Me How: Benefits and Challenges of Agent-Augmented Counterfactual Explanations for Non-Expert Users arXiv: https://arxiv.org/abs/2504.13897 ยท HTML: https://mlsanity.com/paper/2504.13897/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. - Builds on 28 of its 74 references that are in the map. - First posted on arXiv 6 Apr 2025. ## Placed in - LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/) ## Builds on (most cited first, 28 of 28) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023) - [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017) - [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017) - [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023) - [Actionable Recourse in Linear Classification](https://mlsanity.com/paper/1809.06514/) (2018) - [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018) - [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://mlsanity.com/paper/2212.10001/) (2022) - [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023) - [Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers](https://mlsanity.com/paper/1912.03277/) (2019) - [Efficient Search for Diverse Coherent Explanations](https://mlsanity.com/paper/1901.04909/) (2019) - [If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques](https://mlsanity.com/paper/2103.01035/) (2021) - [TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations](https://mlsanity.com/paper/2207.04154/) (2022) - [ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing](https://mlsanity.com/paper/2305.09770/) (2023) - [Properties and Challenges of LLM-Generated Explanations](https://mlsanity.com/paper/2402.10532/) (2024) - [Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorations](https://mlsanity.com/paper/2302.10671/) (2023) - [Zero-shot LLM-guided Counterfactual Generation: A Case Study on NLP Model Evaluation](https://mlsanity.com/paper/2405.04793/) (2024) - [Directive Explanations for Actionable Explainability in Machine Learning Applications](https://mlsanity.com/paper/2102.02671/) (2021) - [From Black Boxes to Conversations: Incorporating XAI in a Conversational Agent](https://mlsanity.com/paper/2209.02552/) (2022) - [EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurations](https://mlsanity.com/paper/2402.00491/) (2024) - [Towards Directive Explanations: Crafting Explainable AI Systems for Actionable Human-AI Interactions](https://mlsanity.com/paper/2401.04118/) (2023) - [Using LLMs for Explaining Sets of Counterfactual Examples to Final Users](https://mlsanity.com/paper/2408.15133/) (2024) - [Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion](https://mlsanity.com/paper/2402.12195/) (2024) - [Explanatory Debiasing: Involving Domain Experts in the Data Generation Process to Mitigate Representation Bias in AI Systems](https://mlsanity.com/paper/2501.01441/) (2024) - [Lessons Learned from EXMOS User Studies: A Technical Report Summarizing Key Takeaways from User Studies Conducted to Evaluate The EXMOS Platform](https://mlsanity.com/paper/2310.02063/) (2023) - [An Explanatory Model Steering System for Collaboration between Domain Experts and AI](https://mlsanity.com/paper/2405.13038/) (2024) - [Representation Debiasing of Generated Data Involving Domain Experts](https://mlsanity.com/paper/2407.09485/) (2024) - [Providing personalized Explanations: a Conversational Approach](https://mlsanity.com/paper/2307.11452/) (2023) ## Built on by (most cited first, 2 of 2) - [Exploring the Impact of Explainable AI and Cognitive Capabilities on Users' Decisions](https://mlsanity.com/paper/2505.01192/) (2025) - [DCFO: Density-Based Counterfactuals for Outliers -- Additional Material](https://mlsanity.com/paper/2512.10659/) (2025) ## Abstract (arXiv) Counterfactual explanations offer actionable insights by illustrating how changes to inputs can lead to different outcomes. However, these explanations often suffer from ambiguity and impracticality, limiting their utility for non-expert users with limited AI knowledge. Augmenting counterfactual explanations with Large Language Models (LLMs) has been proposed as a solution, but little research has examined their benefits and challenges for non-experts. To address this gap, we developed a healthcare-focused system that leverages conversational AI agents to enhance counterfactual explanations, offering clear, actionable recommendations to help patients at high risk of cardiovascular disease (CVD) reduce their risk. Evaluated through a mixed-methods study with 34 participants, our findings highlight the effectiveness of agent-augmented counterfactuals in improving actionable recommendations. Results further indicate that users with prior experience using conversational AI demonstrated greater effectiveness in utilising these explanations compared to novices. Furthermore, this paper introduces a set of generic guidelines for creating augmented counterfactual explanations, incorporating safeguards to mitigate common LLM pitfalls, such as hallucinations, and ensuring the explanations are both actionable and contextually relevant for non-expert users.