# Feature salience - not task-informativeness - drives machine learning model explanations arXiv: https://arxiv.org/abs/2602.09238 · HTML: https://mlsanity.com/paper/2602.09238/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security. - Builds on 26 of its 56 references that are in the map. - First posted on arXiv 9 Feb 2026. ## Placed in - LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/) ## Builds on (most cited first, 26 of 26) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017) - ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016) - [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017) - [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013) - [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019) - [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020) - [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020) - [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019) - [Actionable Recourse in Linear Classification](https://mlsanity.com/paper/1809.06514/) (2018) - [Algorithmic Recourse: from Counterfactual Explanations to Interventions](https://mlsanity.com/paper/2002.06278/) (2020) - [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019) - [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019) - [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019) - [Drug discovery with explainable artificial intelligence](https://mlsanity.com/paper/2007.00523/) (2020) - [Learning how to explain neural networks: PatternNet and PatternAttribution](https://mlsanity.com/paper/1705.05598/) (2017) - [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022) - [The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus](https://mlsanity.com/paper/2302.07265/) (2023) - [Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks](https://mlsanity.com/paper/2401.06654/) (2024) - [Scrutinizing XAI using linear ground-truth data with suppressor variables](https://mlsanity.com/paper/2111.07473/) (2021) - [Theoretical Behavior of XAI Methods in the Presence of Suppressor Variables](https://mlsanity.com/paper/2306.01464/) (2023) - [Optimizing Explanations by Network Canonization and Hyperparameter Search](https://mlsanity.com/paper/2211.17174/) (2022) - [Causal interpretation rules for encoding and decoding models in neuroimaging](https://mlsanity.com/paper/1511.04780/) (2015) - [Benchmarking the Influence of Pre-training on Explanation Performance in MR Image Classification](https://mlsanity.com/paper/2306.12150/) (2023) - [GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations](https://mlsanity.com/paper/2406.11547/) (2024) - [Explainable AI Methods for Neuroimaging: Systematic Failures of Common Tools, the Need for Domain-Specific Validation, and a Proposal for Safe Application](https://mlsanity.com/paper/2508.02560/) (2025) ## Built on by (most cited first, 2 of 2) - [cc-Shapley: Measuring Multivariate Feature Importance Needs Causal Context](https://mlsanity.com/paper/2602.20396/) (2026) - [Local Label-Informed Feature Transfer for Generating Ground-Truth Medical Images: A Comparison of GAN- and Diffusion-Based Approaches](https://mlsanity.com/paper/2607.18882/) (2026) ## Abstract (arXiv) Explainable AI (XAI) promises to provide insight into machine learning models' decision processes, where one goal is to identify failures such as shortcut learning. This promise relies on the field's assumption that input features marked as important by an XAI must contain information about the target variable. However, it is unclear whether informativeness is indeed the main driver of importance attribution in practice, or if other data properties such as statistical suppression, novelty at test-time, or high feature salience substantially contribute. To clarify this, we trained deep learning models on three variants of a binary image classification task, in which translucent watermarks are either absent, act as class-dependent confounds, or represent class-independent noise. Results for five popular attribution methods show substantially elevated relative importance in watermarked areas (RIW) for all models regardless of the training setting (R2 ≥ .45). By contrast, whether the presence of watermarks is class-dependent or not only has a marginal effect on RIW (R2 ≤ .03), despite a clear impact impact on model performance and generalisation ability. XAI methods show similar behaviour to model-agnostic edge detection filters and attribute substantially less importance to watermarks when bright image intensities are encoded by smaller instead of larger feature values. These results indicate that importance attribution is most strongly driven by the salience of image structures at test time rather than statistical associations learned by machine learning models. Previous studies demonstrating successful XAI application should be reevaluated with respect to a possibly spurious concurrency of feature salience and informativeness, and workflows using feature attribution methods as building blocks should be scrutinised.