# Explanations of Black-Box Models based on Directional Feature Interactions

arXiv: https://arxiv.org/abs/2304.07670 · HTML: https://mlsanity.com/paper/2304.07670/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 81 references that are in the map.
- First posted on arXiv 16 Apr 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [GNNExplainer: Generating Explanations for Graph Neural Networks](https://mlsanity.com/paper/1903.03894/) (2019)
- [Parameterized Explainer for Graph Neural Network](https://mlsanity.com/paper/2011.04573/) (2020)
- [Explainability in Graph Neural Networks: A Taxonomic Survey](https://mlsanity.com/paper/2012.15445/) (2020)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [On Explainability of Graph Neural Networks via Subgraph Explorations](https://mlsanity.com/paper/2102.05152/) (2021)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)
- [The Shapley Taylor Interaction Index](https://mlsanity.com/paper/1902.05622/) (2019)
- [Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability](https://mlsanity.com/paper/1910.06358/) (2019)
- [Explaining Explanations: Axiomatic Feature Interactions for Deep Networks](https://mlsanity.com/paper/2002.04138/) (2020)
- [CXPlain: Causal Explanations for Model Interpretation under Uncertainty](https://mlsanity.com/paper/1910.12336/) (2019)
- [Detecting Statistical Interactions from Neural Network Weights](https://mlsanity.com/paper/1705.04977/) (2017)
- [How does this interaction affect me? Interpretable attribution for feature interactions](https://mlsanity.com/paper/2006.10965/) (2020)
- [GraphSVX: Shapley Value Explanations for Graph Neural Networks](https://mlsanity.com/paper/2104.10482/) (2021)
- [Shapley Flow: A Graph-based Approach to Interpreting Model Predictions](https://mlsanity.com/paper/2010.14592/) (2020)
- [Feature Removal Is a Unifying Principle for Model Explanation Methods](https://mlsanity.com/paper/2011.03623/) (2020)
- [Can I trust you more? Model-Agnostic Hierarchical Explanations](https://mlsanity.com/paper/1812.04801/) (2018)

## Built on by (most cited first, 15 of 15)

- [Exact Shapley Attributions in Quadratic-time for FANOVA Gaussian Processes](https://mlsanity.com/paper/2508.14499/) (2025)
- [SmoothHess: ReLU Network Feature Interactions via Stein's Lemma](https://mlsanity.com/paper/2311.00858/) (2023)
- [Evaluating Input Feature Explanations through a Unified Diagnostic Evaluation Framework](https://mlsanity.com/paper/2406.15085/) (2024)
- [Amortized Linear-time Exact Shapley Value for Product-Kernel Methods](https://mlsanity.com/paper/2505.16516/) (2025)
- [Distributing Synergy Functions: Unifying Game-Theoretic Interaction Methods for Machine-Learning Explainability](https://mlsanity.com/paper/2305.03100/) (2023)
- [META-ANOVA: Screening interactions for interpretable machine learning](https://mlsanity.com/paper/2408.00973/) (2024)
- [Axiomatic Explainer Globalness via Optimal Transport](https://mlsanity.com/paper/2411.01126/) (2024)
- [Asymmetric feature interaction for interpreting model predictions](https://mlsanity.com/paper/2305.07224/) (2023)
- [Advancing Biomedicine with Graph Representation Learning: Recent Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2306.10456/) (2023)
- [Explaining Interactions Between Text Spans](https://mlsanity.com/paper/2310.13506/) (2023)
- [DeLELSTM: Decomposition-based Linear Explainable LSTM to Capture Instantaneous and Long-term Effects in Time Series](https://mlsanity.com/paper/2308.13797/) (2023)
- [PE: A Poincare Explanation Method for Fast Text Hierarchy Generation](https://mlsanity.com/paper/2403.16554/) (2024)
- [Exploring Theory-Laden Observations in the Brain Basis of Emotional Experience](https://mlsanity.com/paper/2507.00320/) (2025)
- [OrdShap: Feature Position Importance for Sequential Black-Box Models](https://mlsanity.com/paper/2507.11855/) (2025)
- [OMEGA: Optimizing Machine Learning by Evaluating Generated Algorithms](https://mlsanity.com/paper/2604.26211/) (2026)

## Abstract (arXiv)

As machine learning algorithms are deployed ubiquitously to a variety of domains, it is imperative to make these often black-box models transparent. Several recent works explain black-box models by capturing the most influential features for prediction per instance; such explanation methods are univariate, as they characterize importance per feature. We extend univariate explanation to a higher-order; this enhances explainability, as bivariate methods can capture feature interactions in black-box models, represented as a directed graph. Analyzing this graph enables us to discover groups of features that are equally important (i.e., interchangeable), while the notion of directionality allows us to identify the most influential features. We apply our bivariate method on Shapley value explanations, and experimentally demonstrate the ability of directional explanations to discover feature interactions. We show the superiority of our method against state-of-the-art on CIFAR10, IMDB, Census, Divorce, Drug, and gene data.