# Explaining Reinforcement Learning with Shapley Values

arXiv: https://arxiv.org/abs/2306.05810 · HTML: https://mlsanity.com/paper/2306.05810/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 20 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Builds on (most cited first, 4 of 4)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Shapley explainability on the data manifold](https://mlsanity.com/paper/2006.01272/) (2020)
- [groupShapley: Efficient prediction explanation with Shapley values for feature groups](https://mlsanity.com/paper/2106.12228/) (2021)
- [Causal versus Marginal Shapley Values for Robotic Lever Manipulation Controlled using Deep Reinforcement Learning](https://mlsanity.com/paper/2111.02936/) (2021)

## Built on by (most cited first, 26 of 26)

- [Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans](https://mlsanity.com/paper/2310.19427/) (2023)
- [Explaining Bayesian Optimization by Shapley Values Facilitates Human-AI Collaboration](https://mlsanity.com/paper/2403.04629/) (2024)
- [Towards Multi-dimensional Explanation Alignment for Medical Classification](https://mlsanity.com/paper/2410.21494/) (2024)
- [Interpretability in Action: Exploratory Analysis of VPT, a Minecraft Agent](https://mlsanity.com/paper/2407.12161/) (2024)
- [A Comprehensive Study of Shapley Value in Data Analytics](https://mlsanity.com/paper/2412.01460/) (2024)
- [Behaviour Discovery and Attribution for Explainable Reinforcement Learning](https://mlsanity.com/paper/2503.14973/) (2025)
- [Explaining Reinforcement Learning: A Counterfactual Shapley Values Approach](https://mlsanity.com/paper/2408.02529/) (2024)
- [Why the Agent Made that Decision: Contrastive Explanation Learning for Reinforcement Learning](https://mlsanity.com/paper/2411.16120/) (2024)
- [Towards Automated Semantic Interpretability in Reinforcement Learning via Vision-Language Models](https://mlsanity.com/paper/2503.16724/) (2025)
- [An Explainable AI Framework for Dynamic Resource Management in Vehicular Network Slicing](https://mlsanity.com/paper/2506.11882/) (2025)
- [Interpret Policies in Deep Reinforcement Learning using SILVER with RL-Guided Labeling: A Model-level Approach to High-dimensional and Multi-action Environments](https://mlsanity.com/paper/2510.19244/) (2025)
- [Approximating Shapley Explanations in Reinforcement Learning](https://mlsanity.com/paper/2511.06094/) (2025)
- [Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition](https://mlsanity.com/paper/2603.12046/) (2026)
- [BXRL: Behavior-Explainable Reinforcement Learning](https://mlsanity.com/paper/2603.23738/) (2026)
- [RREDCoT: Segment-Level Reward Redistribution for Reasoning Models](https://mlsanity.com/paper/2606.06475/) (2026)
- [Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.15652/) (2025)
- [Self-Explaining Reinforcement Learning for Mobile Network Resource Allocation](https://mlsanity.com/paper/2509.14925/) (2025)
- [Benchmarking Offline Multi-Objective Reinforcement Learning in Critical Care](https://mlsanity.com/paper/2512.08012/) (2025)
- [Trustworthy and Explainable Deep Reinforcement Learning for Safe and Energy-Efficient Process Control: A Use Case in Industrial Compressed Air Systems](https://mlsanity.com/paper/2512.18317/) (2025)
- [CoMMa: Contribution-Aware Medical Multi-Agents for Decentralized Oncology Decision Support](https://mlsanity.com/paper/2602.09159/) (2026)
- [Enhancing RL Generalizability in Robotics through SHAP Analysis of Algorithms and Hyperparameters](https://mlsanity.com/paper/2605.02867/) (2026)
- [Market Regime Council for Dynamic Credit Assignment in Multi-Agent LLM Decision Systems](https://mlsanity.com/paper/2605.24490/) (2026)
- [Explainable Data-driven Deep Reinforcement Learning Methods for Optimal Energy Management in Buildings](https://mlsanity.com/paper/2606.02049/) (2026)
- [Dueling Deep Q-Learning for Intrusion Detection](https://mlsanity.com/paper/2608.11291/) (2026)
- [Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents](https://mlsanity.com/paper/2608.17524/) (2026)
- [IMPACT-VLA: Interaction-aware Multimodal Propagation Attribution via Counterfactual Trajectories for Vision-Language-Action Policies](https://mlsanity.com/paper/2609.15005/) (2026)

## Abstract (arXiv)

For reinforcement learning systems to be widely adopted, their users must understand and trust them. We present a theoretical analysis of explaining reinforcement learning using Shapley values, following a principled approach from game theory for identifying the contribution of individual players to the outcome of a cooperative game. We call this general framework Shapley Values for Explaining Reinforcement Learning (SVERL). Our analysis exposes the limitations of earlier uses of Shapley values in reinforcement learning. We then develop an approach that uses Shapley values to explain agent performance. In a variety of domains, SVERL produces meaningful explanations that match and supplement human intuition.