# Scrutinizing XAI using linear ground-truth data with suppressor variables

arXiv: https://arxiv.org/abs/2111.07473 · HTML: https://mlsanity.com/paper/2111.07473/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 56 references that are in the map.
- First posted on arXiv 14 Nov 2021.

## Builds on (most cited first, 30 of 31)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Multimodal Explanations: Justifying Decisions and Pointing to the Evidence](https://mlsanity.com/paper/1802.08129/) (2018)
- [When Explanations Lie: Why Many Modified BP Attributions Fail](https://mlsanity.com/paper/1912.09818/) (2019)
- [Benchmarking Attribution Methods with Relative Feature Importance](https://mlsanity.com/paper/1907.09701/) (2019)
- [Learning how to explain neural networks: PatternNet and PatternAttribution](https://mlsanity.com/paper/1705.05598/) (2017)
- [On quantitative aspects of model interpretability](https://mlsanity.com/paper/2007.07584/) (2020)
- [iNNvestigate neural networks!](https://mlsanity.com/paper/1808.04260/) (2018)
- [Quantifying Interpretability and Trust in Machine Learning Systems](https://mlsanity.com/paper/1901.08558/) (2019)
- [Towards Robust Explanations for Deep Neural Networks](https://mlsanity.com/paper/2012.10425/) (2020)
- [Human-in-the-Loop Interpretability Prior](https://mlsanity.com/paper/1805.11571/) (2018)

## Built on by (most cited first, 18 of 18)

- [Theoretical Behavior of XAI Methods in the Presence of Suppressor Variables](https://mlsanity.com/paper/2306.01464/) (2023)
- [Benchmarking the Influence of Pre-training on Explanation Performance in MR Image Classification](https://mlsanity.com/paper/2306.12150/) (2023)
- [XAI-TRIS: Non-linear image benchmarks to quantify false positive post-hoc attribution of feature importance](https://mlsanity.com/paper/2306.12816/) (2023)
- [GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations](https://mlsanity.com/paper/2406.11547/) (2024)
- [Explainable AI needs formalization](https://mlsanity.com/paper/2409.14590/) (2024)
- [EXACT: Towards a platform for empirically benchmarking Machine Learning model explanation methods](https://mlsanity.com/paper/2405.12261/) (2024)
- [Feature salience - not task-informativeness - drives machine learning model explanations](https://mlsanity.com/paper/2602.09238/) (2026)
- [Lost in Context: The Influence of Context on Feature Attribution Methods for Object Recognition](https://mlsanity.com/paper/2411.02833/) (2024)
- [Minimizing False-Positive Attributions in Explanations of Non-Linear Models](https://mlsanity.com/paper/2505.11210/) (2025)
- [Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals](https://mlsanity.com/paper/2602.00171/) (2026)
- [The effect of whitening on explanation performance](https://mlsanity.com/paper/2602.09278/) (2026)
- [Black Box Model Explanations and the Human Interpretability Expectations -- An Analysis in the Context of Homicide Prediction](https://mlsanity.com/paper/2210.10849/) (2022)
- [A Lightweight Generative Model for Interpretable Subject-level Prediction](https://mlsanity.com/paper/2306.11107/) (2023)
- [What's meant by explainable model: A Scoping Review](https://mlsanity.com/paper/2307.09673/) (2023)
- [CHAOSMINING: Benchmarking Post-Hoc Attribution with Sparse Informative Features in High Dimensions](https://mlsanity.com/paper/2406.12150/) (2024)
- [Diagnosing Heteroskedasticity and Resolving Multicollinearity Paradoxes in Physicochemical Property Prediction](https://mlsanity.com/paper/2512.24643/) (2025)
- [cc-Shapley: Measuring Multivariate Feature Importance Needs Causal Context](https://mlsanity.com/paper/2602.20396/) (2026)
- [Local Label-Informed Feature Transfer for Generating Ground-Truth Medical Images: A Comparison of GAN- and Diffusion-Based Approaches](https://mlsanity.com/paper/2607.18882/) (2026)

## Abstract (arXiv)

Machine learning (ML) is increasingly often used to inform high-stakes decisions. As complex ML models (e.g., deep neural networks) are often considered black boxes, a wealth of procedures has been developed to shed light on their inner workings and the ways in which their predictions come about, defining the field of 'explainable AI' (XAI). Saliency methods rank input features according to some measure of 'importance'. Such methods are difficult to validate since a formal definition of feature importance is, thus far, lacking. It has been demonstrated that some saliency methods can highlight features that have no statistical association with the prediction target (suppressor variables). To avoid misinterpretations due to such behavior, we propose the actual presence of such an association as a necessary condition and objective preliminary definition for feature importance. We carefully crafted a ground-truth dataset in which all statistical dependencies are well-defined and linear, serving as a benchmark to study the problem of suppressor variables. We evaluate common explanation methods including LRP, DTD, PatternNet, PatternAttribution, LIME, Anchors, SHAP, and permutation-based methods with respect to our objective definition. We show that most of these methods are unable to distinguish important features from suppressors in this setting.