# Enhancing the Interpretability of SHAP Values Using Large Language Models

arXiv: https://arxiv.org/abs/2409.00079 · HTML: https://mlsanity.com/paper/2409.00079/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2024.

## Built on by (most cited first, 5 of 5)

- [Measuring What LLMs Think They Do: SHAP Faithfulness and Deployability on Financial Tabular Classification](https://mlsanity.com/paper/2512.00163/) (2025)
- [A Two-Stage LLM Framework for Accessible and Verified XAI Explanations](https://mlsanity.com/paper/2604.12543/) (2026)
- [Generative Explainability for Next-Generation Networks: LLM-Augmented XAI with Mutual Feature Interactions](https://mlsanity.com/paper/2606.10942/) (2026)
- [Beyond Ranking Accuracy: Evaluating LLM-Cited Feature Rationales for Next Basket Repurchase Recommendation](https://mlsanity.com/paper/2608.30333/) (2026)
- [Reasoning Externalization for Faithful Large Language Model Narratives of Stock Return Predictions](https://mlsanity.com/paper/2609.38869/) (2026)

## Abstract (arXiv)

Model interpretability is crucial for understanding and trusting the decisions made by complex machine learning models, such as those built with XGBoost. SHAP (SHapley Additive exPlanations) values have become a popular tool for interpreting these models by attributing the output to individual features. However, the technical nature of SHAP explanations often limits their utility to researchers, leaving non-technical end-users struggling to understand the model's behavior. To address this challenge, we explore the use of Large Language Models (LLMs) to translate SHAP value outputs into plain language explanations that are more accessible to non-technical audiences. By applying a pre-trained LLM, we generate explanations that maintain the accuracy of SHAP values while significantly improving their clarity and usability for end users. Our results demonstrate that LLM-enhanced SHAP explanations provide a more intuitive understanding of model predictions, thereby enhancing the overall interpretability of machine learning models. Future work will explore further customization, multimodal explanations, and user feedback mechanisms to refine and expand the approach.