# Shortcut Learning Susceptibility in Vision Classifiers

arXiv: https://arxiv.org/abs/2502.09150 · HTML: https://mlsanity.com/paper/2502.09150/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 15 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Builds on (most cited first, 13 of 13)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Reconstructing Training Data from Trained Neural Networks](https://mlsanity.com/paper/2206.07758/) (2022)
- [On the Foundations of Shortcut Learning](https://mlsanity.com/paper/2310.16228/) (2023)
- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [Autoinverse: Uncertainty Aware Inversion of Neural Networks](https://mlsanity.com/paper/2208.13780/) (2022)
- [Shortcut Learning in Medical Image Segmentation](https://mlsanity.com/paper/2403.06748/) (2024)
- [Landscape Learning for Neural Network Inversion](https://mlsanity.com/paper/2206.09027/) (2022)
- [Rectify ViT Shortcut Learning by Visual Saliency](https://mlsanity.com/paper/2206.08567/) (2022)
- [Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning](https://mlsanity.com/paper/2402.17510/) (2024)
- [Network Inversion of Binarised Neural Nets](https://mlsanity.com/paper/2402.11995/) (2024)
- [Network Inversion of Convolutional Neural Nets](https://mlsanity.com/paper/2407.18002/) (2024)
- [Reconstructing Training Data from Multiclass Neural Networks](https://mlsanity.com/paper/2305.03350/) (2023)
- [Network Inversion for Training-Like Data Reconstruction](https://mlsanity.com/paper/2410.16884/) (2024)

## Built on by (most cited first, 4 of 4)

- [eXplaining to Learn (eX2L): Regularization Using Contrastive Visual Explanation Pairs for Distribution Shifts](https://mlsanity.com/paper/2605.06368/) (2026)
- [VEIL: How Visual Encoding Hijacking Induces Bias In Vision Models](https://mlsanity.com/paper/2607.05641/) (2026)
- [Invisible Shortcuts: Why Vision Encoders Know Your Camera](https://mlsanity.com/paper/2608.05424/) (2026)
- [CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Models](https://mlsanity.com/paper/2609.30979/) (2026)

## Abstract (arXiv)

Shortcut learning, where machine learning models exploit spurious correlations in data instead of capturing meaningful features, poses a significant challenge to building robust and generalizable models. This phenomenon is prevalent across various machine learning applications, including vision, natural language processing, and speech recognition, where models may find unintended cues that minimize training loss but fail to capture the underlying structure of the data. Vision classifiers based on Convolutional Neural Networks (CNNs), Multi-Layer Perceptrons (MLPs), and Vision Transformers (ViTs) leverage distinct architectural principles to process spatial and structural information, making them differently susceptible to shortcut learning. In this study, we systematically evaluate these architectures by introducing deliberate shortcuts into the dataset that are correlated with class labels both positionally and via intensity, creating a controlled setup to assess whether models rely on these artificial cues or learn actual distinguishing features. We perform both quantitative evaluation by training on the shortcut-modified dataset and testing on two different test sets-one containing the same shortcuts and another without them-to determine the extent of reliance on shortcuts. Additionally, qualitative evaluation is performed using network inversion-based reconstruction techniques to analyze what the models internalize in their weights, aiming to reconstruct the training data as perceived by the classifiers. Further, we evaluate susceptibility to shortcut learning across different learning rates. Our analysis reveals that CNNs at lower learning rates tend to be more reserved against entirely picking up shortcut features, while ViTs, particularly those without positional encodings, almost entirely ignore the distinctive image features in the presence of shortcuts.