# Towards Rigorous Interpretations: a Formalisation of Feature Attribution

arXiv: https://arxiv.org/abs/2104.12437 · HTML: https://mlsanity.com/paper/2104.12437/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 26 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)
- [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019)
- [Problems with Shapley-value-based explanations as feature importance measures](https://mlsanity.com/paper/2002.11097/) (2020)
- [When Explanations Lie: Why Many Modified BP Attributions Fail](https://mlsanity.com/paper/1912.09818/) (2019)
- [How does this interaction affect me? Interpretable attribution for feature interactions](https://mlsanity.com/paper/2006.10965/) (2020)
- [Purifying Interaction Effects with the Functional ANOVA: An Efficient Algorithm for Recovering Identifiable Additive Models](https://mlsanity.com/paper/1911.04974/) (2019)
- [Improving performance of deep learning models with axiomatic attribution priors and expected gradients](https://mlsanity.com/paper/1906.10670/) (2019)
- [Feature Removal Is a Unifying Principle for Model Explanation Methods](https://mlsanity.com/paper/2011.03623/) (2020)
- [Challenging common interpretability assumptions in feature attribution explanations](https://mlsanity.com/paper/2012.02748/) (2020)
- [Explaining Deep Neural Networks](https://mlsanity.com/paper/2010.01496/) (2020)
- [Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction](https://mlsanity.com/paper/2008.11406/) (2020)

## Built on by (most cited first, 11 of 11)

- [A Consistent and Efficient Evaluation Strategy for Attribution Methods](https://mlsanity.com/paper/2202.00449/) (2022)
- [Impossibility Theorems for Feature Attribution](https://mlsanity.com/paper/2212.11870/) (2022)
- [The Inadequacy of Shapley Values for Explainability](https://mlsanity.com/paper/2302.08160/) (2023)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [A Refutation of Shapley Values for Explainability](https://mlsanity.com/paper/2309.03041/) (2023)
- [Learning Unsupervised Hierarchies of Audio Concepts](https://mlsanity.com/paper/2207.11231/) (2022)
- [Local Interpretable Model Agnostic Shap Explanations for machine learning models](https://mlsanity.com/paper/2210.04533/) (2022)
- [Logic-Based Explainability: Past, Present & Future](https://mlsanity.com/paper/2406.11873/) (2024)
- [Time-series attribution maps with regularized contrastive learning](https://mlsanity.com/paper/2502.12977/) (2025)
- [Explainability as statistical inference](https://mlsanity.com/paper/2212.03131/) (2022)
- [Prospector Heads: Generalized Feature Attribution for Large Models & Data](https://mlsanity.com/paper/2402.11729/) (2024)

## Abstract (arXiv)

Feature attribution is often loosely presented as the process of selecting a subset of relevant features as a rationale of a prediction. Task-dependent by nature, precise definitions of "relevance" encountered in the literature are however not always consistent. This lack of clarity stems from the fact that we usually do not have access to any notion of ground-truth attribution and from a more general debate on what good interpretations are. In this paper we propose to formalise feature selection/attribution based on the concept of relaxed functional dependence. In particular, we extend our notions to the instance-wise setting and derive necessary properties for candidate selection solutions, while leaving room for task-dependence. By computing ground-truth attributions on synthetic datasets, we evaluate many state-of-the-art attribution methods and show that, even when optimised, some fail to verify the proposed properties and provide wrong solutions.