# LLMs for XAI: Future Directions for Explaining Explanations

arXiv: https://arxiv.org/abs/2405.06064 · HTML: https://mlsanity.com/paper/2405.06064/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 11 of 11)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations](https://mlsanity.com/paper/2207.04154/) (2022)
- [Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era](https://mlsanity.com/paper/2403.08946/) (2024)
- [Towards LLM-guided Causal Explainability for Black-box Text Classifiers](https://mlsanity.com/paper/2309.13340/) (2023)
- [ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing](https://mlsanity.com/paper/2305.09770/) (2023)
- [Sibyl: Understanding and Addressing the Usability Challenges of Machine Learning In High-Stakes Decision Making](https://mlsanity.com/paper/2103.02071/) (2021)
- [From Black Boxes to Conversations: Incorporating XAI in a Conversational Agent](https://mlsanity.com/paper/2209.02552/) (2022)
- [In-Context Explainers: Harnessing LLMs for Explaining Black Box Models](https://mlsanity.com/paper/2310.05797/) (2023)
- [On Two XAI Cultures: A Case Study of Non-technical Explanations in Deployed AI System](https://mlsanity.com/paper/2112.01016/) (2021)
- [Interpretation Quality Score for Measuring the Quality of interpretability methods](https://mlsanity.com/paper/2205.12254/) (2022)

## Built on by (most cited first, 19 of 19)

- [LLMs for Explainable AI: A Comprehensive Survey](https://mlsanity.com/paper/2504.00125/) (2025)
- [Decision Information Meets Large Language Models: The Future of Explainable Operations Research](https://mlsanity.com/paper/2502.09994/) (2025)
- [Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration](https://mlsanity.com/paper/2409.12809/) (2024)
- [Thoughts without Thinking: Reconsidering the Explanatory Value of Chain-of-Thought Reasoning in LLMs through Agentic Pipelines](https://mlsanity.com/paper/2505.00875/) (2025)
- [Explingo: Explaining AI Predictions using Large Language Models](https://mlsanity.com/paper/2412.05145/) (2024)
- [Retrieval Augmented Generation with Multi-Modal LLM Framework for Wireless Environments](https://mlsanity.com/paper/2503.07670/) (2025)
- [An Agentic Approach to Generating XAI-Narratives](https://mlsanity.com/paper/2603.20003/) (2026)
- [Don't Just Translate, Agitate: Using Large Language Models as Devil's Advocates for AI Explanations](https://mlsanity.com/paper/2504.12424/) (2025)
- [A three-Level Framework for LLM-Enhanced eXplainable AI: From technical explanations to natural language](https://mlsanity.com/paper/2506.05887/) (2025)
- [Explain First, Trust Later: LLM-Augmented Explanations for Graph-Based Crypto Anomaly Detection](https://mlsanity.com/paper/2506.14933/) (2025)
- [From XAI to Stories: A Factorial Study of LLM-Generated Explanation Quality](https://mlsanity.com/paper/2601.02224/) (2026)
- [Quality Without Usefulness: LLM-Generated XAI Narratives as Trust Heuristics Rather Than Decision Aids](https://mlsanity.com/paper/2605.26770/) (2026)
- [A Two-Stage LLM Framework for Accessible and Verified XAI Explanations](https://mlsanity.com/paper/2604.12543/) (2026)
- [GUIDEQ: Framework for Guided Questioning for progressive informational collection and classification](https://mlsanity.com/paper/2411.05991/) (2024)
- [ProfileXAI: User-Adaptive Explainable AI](https://mlsanity.com/paper/2510.22998/) (2025)
- [Why is "Chicago" Predictive of Deceptive Reviews? Using LLMs to Discover Language Phenomena from Lexical Cues](https://mlsanity.com/paper/2511.13658/) (2025)
- [Textual Explanations and Their Evaluations for Reinforcement Learning Policy](https://mlsanity.com/paper/2601.02514/) (2026)
- [A Mobile Application Front-End for Presenting Explainable AI Results in Diabetes Risk Estimation](https://mlsanity.com/paper/2601.15292/) (2025)
- [NEURON: A Neuro-symbolic System for Grounded Clinical Explainability](https://mlsanity.com/paper/2605.01189/) (2026)

## Abstract (arXiv)

In response to the demand for Explainable Artificial Intelligence (XAI), we investigate the use of Large Language Models (LLMs) to transform ML explanations into natural, human-readable narratives. Rather than directly explaining ML models using LLMs, we focus on refining explanations computed using existing XAI algorithms. We outline several research directions, including defining evaluation metrics, prompt design, comparing LLM models, exploring further training methods, and integrating external data. Initial experiments and user study suggest that LLMs offer a promising way to enhance the interpretability and usability of XAI.