# XAI-TRIS: Non-linear image benchmarks to quantify false positive post-hoc attribution of feature importance

arXiv: https://arxiv.org/abs/2306.12816 · HTML: https://mlsanity.com/paper/2306.12816/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 22 Jun 2023.

## Builds on (most cited first, 25 of 25)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [The Risks of Invariant Risk Minimization](https://mlsanity.com/paper/2010.05761/) (2020)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI](https://mlsanity.com/paper/2003.07258/) (2020)
- [OpenXAI: Towards a Transparent Evaluation of Model Explanations](https://mlsanity.com/paper/2206.11104/) (2022)
- [Do Feature Attribution Methods Correctly Attribute Features?](https://mlsanity.com/paper/2104.14403/) (2021)
- [Learning how to explain neural networks: PatternNet and PatternAttribution](https://mlsanity.com/paper/1705.05598/) (2017)
- [iNNvestigate neural networks!](https://mlsanity.com/paper/1808.04260/) (2018)
- [PASS: An ImageNet replacement for self-supervised pretraining without humans](https://mlsanity.com/paper/2109.13228/) (2021)
- [Evaluating Feature Attribution Methods in the Image Domain](https://mlsanity.com/paper/2202.12270/) (2022)
- [Scrutinizing XAI using linear ground-truth data with suppressor variables](https://mlsanity.com/paper/2111.07473/) (2021)
- [Theoretical Behavior of XAI Methods in the Presence of Suppressor Variables](https://mlsanity.com/paper/2306.01464/) (2023)
- [Benchmarking the Influence of Pre-training on Explanation Performance in MR Image Classification](https://mlsanity.com/paper/2306.12150/) (2023)

## Built on by (most cited first, 7 of 7)

- [GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations](https://mlsanity.com/paper/2406.11547/) (2024)
- [Explainable AI needs formalization](https://mlsanity.com/paper/2409.14590/) (2024)
- [Minimizing False-Positive Attributions in Explanations of Non-Linear Models](https://mlsanity.com/paper/2505.11210/) (2025)
- [The effect of whitening on explanation performance](https://mlsanity.com/paper/2602.09278/) (2026)
- [AGOP as Explanation: From Feature Learning to Per-Sample Attribution in Image Classifiers](https://mlsanity.com/paper/2605.12816/) (2026)
- [Attribution Upsampling should Redistribute, Not Interpolate](https://mlsanity.com/paper/2603.16067/) (2026)
- [Local Label-Informed Feature Transfer for Generating Ground-Truth Medical Images: A Comparison of GAN- and Diffusion-Based Approaches](https://mlsanity.com/paper/2607.18882/) (2026)

## Abstract (arXiv)

The field of 'explainable' artificial intelligence (XAI) has produced highly cited methods that seek to make the decisions of complex machine learning (ML) methods 'understandable' to humans, for example by attributing 'importance' scores to input features. Yet, a lack of formal underpinning leaves it unclear as to what conclusions can safely be drawn from the results of a given XAI method and has also so far hindered the theoretical verification and empirical validation of XAI methods. This means that challenging non-linear problems, typically solved by deep neural networks, presently lack appropriate remedies. Here, we craft benchmark datasets for three different non-linear classification scenarios, in which the important class-conditional features are known by design, serving as ground truth explanations. Using novel quantitative metrics, we benchmark the explanation performance of a wide set of XAI methods across three deep learning model architectures. We show that popular XAI methods are often unable to significantly outperform random performance baselines and edge detection methods. Moreover, we demonstrate that explanations derived from different model architectures can be vastly different; thus, prone to misinterpretation even under controlled conditions.