# Shapley Flow: A Graph-based Approach to Interpreting Model Predictions

arXiv: https://arxiv.org/abs/2010.14592 · HTML: https://mlsanity.com/paper/2010.14592/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 30 of 51)

- [Algorithms to estimate Shapley value feature attributions](https://mlsanity.com/paper/2207.07605/) (2022)
- [The Shapley Value in Machine Learning](https://mlsanity.com/paper/2202.05594/) (2022)
- [Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applications](https://mlsanity.com/paper/2103.04244/) (2021)
- [Causally motivated Shortcut Removal Using Auxiliary Labels](https://mlsanity.com/paper/2105.06422/) (2021)
- [Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00285/) (2021)
- [Defining and Quantifying the Emergence of Sparse Concepts in DNNs](https://mlsanity.com/paper/2111.06206/) (2021)
- [WeightedSHAP: analyzing and improving Shapley based feature attributions](https://mlsanity.com/paper/2209.13429/) (2022)
- [SHAFF: Fast and consistent SHApley eFfect estimates via random Forests](https://mlsanity.com/paper/2105.11724/) (2021)
- [Counterfactual Shapley Additive Explanations](https://mlsanity.com/paper/2110.14270/) (2021)
- [Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena](https://mlsanity.com/paper/2206.05487/) (2022)
- [Improvement-Focused Causal Recourse (ICR)](https://mlsanity.com/paper/2210.15709/) (2022)
- ["Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts](https://mlsanity.com/paper/2210.10769/) (2022)
- [Explaining Algorithmic Fairness Through Fairness-Aware Causal Path Decomposition](https://mlsanity.com/paper/2108.05335/) (2021)
- [Explanations of Black-Box Models based on Directional Feature Interactions](https://mlsanity.com/paper/2304.07670/) (2023)
- [A Comparative Study of Methods for Estimating Conditional Shapley Values and When to Use Them](https://mlsanity.com/paper/2305.09536/) (2023)
- [Statistical Aspects of SHAP: Functional ANOVA for Model Interpretation](https://mlsanity.com/paper/2208.09970/) (2022)
- [Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning](https://mlsanity.com/paper/2106.02938/) (2021)
- [AcME -- Accelerated Model-agnostic Explanations: Fast Whitening of the Machine-Learning Black Box](https://mlsanity.com/paper/2112.12635/) (2021)
- [Trade-off Between Efficiency and Consistency for Removal-based Explanations](https://mlsanity.com/paper/2210.17426/) (2022)
- [Accurate Explanation Model for Image Classifiers using Class Association Embedding](https://mlsanity.com/paper/2406.07961/) (2024)
- [Practical do-Shapley Explanations with Estimand-Agnostic Causal Inference](https://mlsanity.com/paper/2509.20211/) (2025)
- [Causal Forecasting:Generalization Bounds for Autoregressive Models](https://mlsanity.com/paper/2111.09831/) (2021)
- [Exact Shapley Values for Local and Model-True Explanations of Decision Tree Ensembles](https://mlsanity.com/paper/2112.10592/) (2021)
- [Perception Visualization: Seeing Through the Eyes of a DNN](https://mlsanity.com/paper/2204.09920/) (2022)
- [DiConStruct: Causal Concept-based Explanations through Black-Box Distillation](https://mlsanity.com/paper/2401.08534/) (2024)
- [Efficient SAGE Estimation via Causal Structure Learning](https://mlsanity.com/paper/2304.03113/) (2023)
- [Counterfactual Effect Decomposition in Multi-Agent Sequential Decision Making](https://mlsanity.com/paper/2410.12539/) (2024)
- [Comparing Baseline Shapley and Integrated Gradients for Local Explanation: Some Additional Insights](https://mlsanity.com/paper/2208.06096/) (2022)
- [PWSHAP: A Path-Wise Explanation Model for Targeted Variables](https://mlsanity.com/paper/2306.14672/) (2023)
- [Towards consistency of rule-based explainer and black box model -- fusion of rule induction and XAI-based feature importance](https://mlsanity.com/paper/2407.14543/) (2024)

## Abstract (arXiv)

Many existing approaches for estimating feature importance are problematic because they ignore or hide dependencies among features. A causal graph, which encodes the relationships among input variables, can aid in assigning feature importance. However, current approaches that assign credit to nodes in the causal graph fail to explain the entire graph. In light of these limitations, we propose Shapley Flow, a novel approach to interpreting machine learning models. It considers the entire causal graph, and assigns credit to textitedges instead of treating nodes as the fundamental unit of credit assignment. Shapley Flow is the unique solution to a generalization of the Shapley value axioms to directed acyclic graphs. We demonstrate the benefit of using Shapley Flow to reason about the impact of a model's input on its output. In addition to maintaining insights from existing approaches, Shapley Flow extends the flat, set-based, view prevalent in game theory based explanation methods to a deeper, textitgraph-based, view. This graph-based view enables users to understand the flow of importance through a system, and reason about potential interventions.