# Salient ImageNet: How to discover spurious features in Deep Learning?

arXiv: https://arxiv.org/abs/2110.04301 · HTML: https://mlsanity.com/paper/2110.04301/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 51 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [Inverting Visual Representations with Convolutional Networks](https://mlsanity.com/paper/1506.02753/) (2015)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks](https://mlsanity.com/paper/1602.03616/) (2016)
- [Interpreting Deep Visual Representations via Network Dissection](https://mlsanity.com/paper/1711.05611/) (2017)
- [Benchmarking Deep Learning Interpretability in Time Series Predictions](https://mlsanity.com/paper/2010.13924/) (2020)
- [Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images](https://mlsanity.com/paper/1512.02017/) (2015)
- [From ImageNet to Image Classification: Contextualizing Progress on Benchmarks](https://mlsanity.com/paper/2005.11295/) (2020)

## Built on by (most cited first, 30 of 114)

- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [On Feature Learning in the Presence of Spurious Correlations](https://mlsanity.com/paper/2210.11369/) (2022)
- [Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors](https://mlsanity.com/paper/2403.07366/) (2024)
- [A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others](https://mlsanity.com/paper/2212.04825/) (2022)
- [Discover and Cure: Concept-aware Mitigation of Spurious Correlation](https://mlsanity.com/paper/2305.00650/) (2023)
- [On the Foundations of Shortcut Learning](https://mlsanity.com/paper/2310.16228/) (2023)
- [When does dough become a bagel? Analyzing the remaining mistakes on ImageNet](https://mlsanity.com/paper/2205.04596/) (2022)
- [Discovering and Mitigating Visual Biases through Keyword Explanation](https://mlsanity.com/paper/2301.11104/) (2023)
- [A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes](https://mlsanity.com/paper/2201.10766/) (2022)
- [Towards Last-layer Retraining for Group Robustness with Fewer Annotations](https://mlsanity.com/paper/2309.08534/) (2023)
- [SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2312.00825/) (2023)
- [LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images](https://mlsanity.com/paper/2305.19164/) (2023)
- [ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations](https://mlsanity.com/paper/2211.01866/) (2022)
- [Contrastive language and vision learning of general fashion concepts](https://mlsanity.com/paper/2204.03972/) (2022)
- [Spurious Features Everywhere -- Large-Scale Detection of Harmful Spurious Features in ImageNet](https://mlsanity.com/paper/2212.04871/) (2022)
- [Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases](https://mlsanity.com/paper/2303.05470/) (2023)
- [COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs](https://mlsanity.com/paper/2309.14356/) (2023)
- [Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation](https://mlsanity.com/paper/2508.06426/) (2025)
- [ModelDiff: A Framework for Comparing Learning Algorithms](https://mlsanity.com/paper/2211.12491/) (2022)
- [Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives](https://mlsanity.com/paper/2307.16851/) (2023)
- [Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations](https://mlsanity.com/paper/2403.07241/) (2024)
- [Doubly Right Object Recognition: A Why Prompt for Visual Rationales](https://mlsanity.com/paper/2212.06202/) (2022)
- [FACTS: First Amplify Correlations and Then Slice to Discover Bias](https://mlsanity.com/paper/2309.17430/) (2023)
- [Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features](https://mlsanity.com/paper/2212.13881/) (2022)
- [Explicit Tradeoffs between Adversarial and Natural Distributional Robustness](https://mlsanity.com/paper/2209.07592/) (2022)
- [When Does Group Invariant Learning Survive Spurious Correlations?](https://mlsanity.com/paper/2206.14534/) (2022)
- [Core Risk Minimization using Salient ImageNet](https://mlsanity.com/paper/2203.15566/) (2022)
- [Understanding Visual Feature Reliance through the Lens of Complexity](https://mlsanity.com/paper/2407.06076/) (2024)
- [Adversarial Unlearning: Reducing Confidence Along Adversarial Directions](https://mlsanity.com/paper/2206.01367/) (2022)
- [DASH: Detection and Assessment of Systematic Hallucinations of VLMs](https://mlsanity.com/paper/2503.23573/) (2025)

## Abstract (arXiv)

Deep neural networks can be unreliable in the real world especially when they heavily use it spurious features for their predictions. Focusing on image classifications, we define it core features as the set of visual features that are always a part of the object definition while it spurious features are the ones that are likely to it co-occur with the object but not a part of it (e.g., attribute "fingers" for class "band aid"). Traditional methods for discovering spurious features either require extensive human annotations (thus, not scalable), or are useful on specific models. In this work, we introduce a it general framework to discover a subset of spurious and core visual features used in inferences of a general model and localize them on a large number of images with minimal human supervision. Our methodology is based on this key idea: to identify spurious or core textitvisual features used in model predictions, we identify spurious or core textitneural features (penultimate layer neurons of a robust model) via limited human supervision (e.g., using top 5 activating images per feature). We then show that these neural feature annotations it generalize extremely well to many more images it without any human supervision. We use the activation maps for these neural features as the soft masks to highlight spurious or core visual features. Using this methodology, we introduce the it Salient Imagenet dataset containing core and spurious masks for a large set of samples from Imagenet. Using this dataset, we show that several popular Imagenet models rely heavily on various spurious features in their predictions, indicating the standard accuracy alone is not sufficient to fully assess model performance. Code and dataset for reproducing all experiments in the paper is available at urlhttps://github.com/singlasahil14/salientimagenet.