# TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation

arXiv: https://arxiv.org/abs/2407.10114 · HTML: https://mlsanity.com/paper/2407.10114/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 14 Jul 2024.

## Builds on (most cited first, 12 of 12)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [A Survey of the State of Explainable AI for Natural Language Processing](https://mlsanity.com/paper/2010.00711/) (2020)
- [L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data](https://mlsanity.com/paper/1808.02610/) (2018)
- [NILE : Natural Language Inference with Faithful Natural Language Explanations](https://mlsanity.com/paper/2005.12116/) (2020)

## Built on by (most cited first, 30 of 42)

- [Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning](https://mlsanity.com/paper/2506.02867/) (2025)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [SCAR: Shapley Credit Assignment for More Efficient RLHF](https://mlsanity.com/paper/2505.20417/) (2025)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Causality Matters: How Temporal Information Emerges in Video Language Models](https://mlsanity.com/paper/2508.11576/) (2025)
- [Attention, Please! PixelSHAP Reveals What Vision-Language Models Actually Focus On](https://mlsanity.com/paper/2503.06670/) (2025)
- [MultiSHAP: A Shapley-Based Framework for Explaining Cross-Modal Interactions in Multimodal AI Models](https://mlsanity.com/paper/2508.00576/) (2025)
- [BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principle](https://mlsanity.com/paper/2502.18331/) (2025)
- [Concept-Level Explainability for Auditing & Steering LLM Responses](https://mlsanity.com/paper/2505.07610/) (2025)
- [Can Global XAI Methods Reveal Injected Behaviours in LLMs? SHAP vs Rule Extraction vs RuleSHAP](https://mlsanity.com/paper/2505.11189/) (2025)
- [TAG-INSTRUCT: Controlled Instruction Complexity Enhancement through Structure-based Augmentation](https://mlsanity.com/paper/2505.18557/) (2025)
- [Anchors in the Machine: Behavioral and Attributional Evidence of Anchoring Bias in LLMs](https://mlsanity.com/paper/2511.05766/) (2025)
- [Measuring What LLMs Think They Do: SHAP Faithfulness and Deployability on Financial Tabular Classification](https://mlsanity.com/paper/2512.00163/) (2025)
- [Towards Reliable and Interpretable Traffic Crash Pattern Prediction and Safety Interventions Using Customized Large Language Models](https://mlsanity.com/paper/2505.12545/) (2025)
- [llmSHAP: A Principled Approach to LLM Explainability](https://mlsanity.com/paper/2511.01311/) (2025)
- [A Decentralized Retrieval Augmented Generation System with Source Reliabilities Secured on Blockchain](https://mlsanity.com/paper/2511.07577/) (2025)
- [Quantifying Modality Contributions via Disentangling Multimodal Representations](https://mlsanity.com/paper/2511.19470/) (2025)
- [Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization](https://mlsanity.com/paper/2512.15765/) (2025)
- [Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMs](https://mlsanity.com/paper/2601.08403/) (2026)
- [Interpretable Probability Estimation with LLMs via Shapley Reconstruction](https://mlsanity.com/paper/2601.09151/) (2026)
- [ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Training](https://mlsanity.com/paper/2603.29871/) (2026)
- [AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Training](https://mlsanity.com/paper/2606.01635/) (2026)
- [SkillShapley: Boundary-Adaptive Shapley Valuation for Skill Step Attribution in LLM Agents](https://mlsanity.com/paper/2608.13173/) (2026)
- [Concept Based Continuous Prompts for Interpretable Text Classification](https://mlsanity.com/paper/2412.01644/) (2024)
- [ExpertLens: Activation steering features are highly interpretable](https://mlsanity.com/paper/2502.15090/) (2025)
- [PLEX: Perturbation-free Local Explanations for LLM-Based Text Classification](https://mlsanity.com/paper/2507.10596/) (2025)
- [Explaining Fine Tuned LLMs via Counterfactuals A Knowledge Graph Driven Framework](https://mlsanity.com/paper/2509.21241/) (2025)
- [A Video Is Not Worth a Thousand Words](https://mlsanity.com/paper/2510.23253/) (2025)
- [Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian](https://mlsanity.com/paper/2511.19719/) (2025)
- [AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation](https://mlsanity.com/paper/2512.12597/) (2025)

## Abstract (arXiv)

As large language models (LLMs) become increasingly prevalent in critical applications, the need for interpretable AI has grown. We introduce TokenSHAP, a novel method for interpreting LLMs by attributing importance to individual tokens or substrings within input prompts. This approach adapts Shapley values from cooperative game theory to natural language processing, offering a rigorous framework for understanding how different parts of an input contribute to a model's response. TokenSHAP leverages Monte Carlo sampling for computational efficiency, providing interpretable, quantitative measures of token importance. We demonstrate its efficacy across diverse prompts and LLM architectures, showing consistent improvements over existing baselines in alignment with human judgments, faithfulness to model behavior, and consistency. Our method's ability to capture nuanced interactions between tokens provides valuable insights into LLM behavior, enhancing model transparency, improving prompt engineering, and aiding in the development of more reliable AI systems. TokenSHAP represents a significant step towards the necessary interpretability for responsible AI deployment, contributing to the broader goal of creating more transparent, accountable, and trustworthy AI systems.