# Explaining Explanations: Axiomatic Feature Interactions for Deep Networks

arXiv: https://arxiv.org/abs/2002.04138 · HTML: https://mlsanity.com/paper/2002.04138/

## Key facts

- Cited by 103 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2020.

## Built on by (most cited first, 30 of 103)

- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [Faith-Shap: The Faithful Shapley Interaction Index](https://mlsanity.com/paper/2203.00870/) (2022)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [SHAP-IQ: Unified Approximation of any-order Shapley Interactions](https://mlsanity.com/paper/2303.01179/) (2023)
- [Discovering and Explaining the Representation Bottleneck of DNNs](https://mlsanity.com/paper/2111.06236/) (2021)
- [Augmenting Interpretable Models with LLMs during Training](https://mlsanity.com/paper/2209.11799/) (2022)
- [Order in the Court: Explainable AI Methods Prone to Disagreement](https://mlsanity.com/paper/2105.03287/) (2021)
- [Defining and Quantifying the Emergence of Sparse Concepts in DNNs](https://mlsanity.com/paper/2111.06206/) (2021)
- [Gradient based Feature Attribution in Explainable AI: A Technical Review](https://mlsanity.com/paper/2403.10415/) (2024)
- [Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selection](https://mlsanity.com/paper/2209.09326/) (2022)
- [Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks](https://mlsanity.com/paper/2401.06654/) (2024)
- [Does a Neural Network Really Encode Symbolic Concepts?](https://mlsanity.com/paper/2302.13080/) (2023)
- [AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression](https://mlsanity.com/paper/2305.10010/) (2023)
- [Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces](https://mlsanity.com/paper/2212.14855/) (2022)
- [KernelSHAP-IQ: Weighted Least-Square Optimization for Shapley Interactions](https://mlsanity.com/paper/2405.10852/) (2024)
- [Algorithm Fairness in AI for Medicine and Healthcare](https://mlsanity.com/paper/2110.00603/) (2021)
- [Concept Embedding Analysis: A Review](https://mlsanity.com/paper/2203.13909/) (2022)
- [PredDiff: Explanations and Interactions from Conditional Expectations](https://mlsanity.com/paper/2102.13519/) (2021)
- [A Unified Game-Theoretic Interpretation of Adversarial Robustness](https://mlsanity.com/paper/2111.03536/) (2021)
- [MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset](https://mlsanity.com/paper/2102.06761/) (2021)
- [Explanations of Black-Box Models based on Directional Feature Interactions](https://mlsanity.com/paper/2304.07670/) (2023)
- [Interpreting Attributions and Interactions of Adversarial Attacks](https://mlsanity.com/paper/2108.06895/) (2021)
- [Proving Common Mechanisms Shared by Twelve Methods of Boosting Adversarial Transferability](https://mlsanity.com/paper/2207.11694/) (2022)
- [Interpretable Text Embeddings and Text Similarity Explanation: A Survey](https://mlsanity.com/paper/2502.14862/) (2025)
- [NeuroX Library for Neuron Analysis of Deep NLP Models](https://mlsanity.com/paper/2305.17073/) (2023)
- [Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?](https://mlsanity.com/paper/2105.10719/) (2021)
- [Explaining Predictive Uncertainty by Exposing Second-Order Effects](https://mlsanity.com/paper/2401.17441/) (2024)
- [Unifying Feature-Based Explanations with Functional ANOVA and Cooperative Game Theory](https://mlsanity.com/paper/2412.17152/) (2024)
- [Bayesian Neural Networks Avoid Encoding Complex and Perturbation-Sensitive Concepts](https://mlsanity.com/paper/2302.13095/) (2023)

## Abstract (arXiv)

Recent work has shown great promise in explaining neural network behavior. In particular, feature attribution methods explain which features were most important to a model's prediction on a given input. However, for many tasks, simply knowing which features were important to a model's prediction may not provide enough insight to understand model behavior. The interactions between features within the model may better help us understand not only the model, but also why certain features are more important than others. In this work, we present Integrated Hessians, an extension of Integrated Gradients that explains pairwise feature interactions in neural networks. Integrated Hessians overcomes several theoretical limitations of previous methods to explain interactions, and unlike such previous methods is not limited to a specific architecture or class of neural network. Additionally, we find that our method is faster than existing methods when the number of features is large, and outperforms previous methods on existing quantitative benchmarks. Code available at https://github.com/suinleelab/path_explain