# Automatic Shortcut Removal for Self-Supervised Representation Learning

arXiv: https://arxiv.org/abs/2002.08822 · HTML: https://mlsanity.com/paper/2002.08822/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2020.

## Built on by (most cited first, 30 of 43)

- [Learning Debiased Representation via Disentangled Feature Augmentation](https://mlsanity.com/paper/2107.01372/) (2021)
- [Can contrastive learning avoid shortcut solutions?](https://mlsanity.com/paper/2106.11230/) (2021)
- [Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models](https://mlsanity.com/paper/2110.07736/) (2021)
- [On the Foundations of Shortcut Learning](https://mlsanity.com/paper/2310.16228/) (2023)
- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [Explore Spurious Correlations at the Concept Level in Language Models for Text Classification](https://mlsanity.com/paper/2311.08648/) (2023)
- [ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks](https://mlsanity.com/paper/2201.02263/) (2022)
- [Geographical Knowledge-driven Representation Learning for Remote Sensing Images](https://mlsanity.com/paper/2107.05276/) (2021)
- [Object-aware Contrastive Learning for Debiased Scene Representation](https://mlsanity.com/paper/2108.00049/) (2021)
- [Avoiding spurious correlations via logit correction](https://mlsanity.com/paper/2212.01433/) (2022)
- [Domain Generalization by Learning and Removing Domain-specific Features](https://mlsanity.com/paper/2212.07101/) (2022)
- [What do neural networks learn in image classification? A frequency shortcut perspective](https://mlsanity.com/paper/2307.09829/) (2023)
- [Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data](https://mlsanity.com/paper/2206.02353/) (2022)
- [Group Robust Classification Without Any Group Information](https://mlsanity.com/paper/2310.18555/) (2023)
- [A Survey on the Robustness of Computer Vision Models against Common Corruptions](https://mlsanity.com/paper/2305.06024/) (2023)
- [Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning](https://mlsanity.com/paper/2205.12466/) (2022)
- [Parametric Scattering Networks](https://mlsanity.com/paper/2107.09539/) (2021)
- [Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation](https://mlsanity.com/paper/2210.04497/) (2022)
- [Latent-OFER: Detect, Mask, and Reconstruct with Latent Vectors for Occluded Facial Expression Recognition](https://mlsanity.com/paper/2307.11404/) (2023)
- [DiagViB-6: A Diagnostic Benchmark Suite for Vision Models in the Presence of Shortcut and Generalization Opportunities](https://mlsanity.com/paper/2108.05779/) (2021)
- [When does Privileged Information Explain Away Label Noise?](https://mlsanity.com/paper/2303.01806/) (2023)
- [Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning](https://mlsanity.com/paper/2212.08378/) (2022)
- [A Too-Good-to-be-True Prior to Reduce Shortcut Reliance](https://mlsanity.com/paper/2102.06406/) (2021)
- [VOLTA: an Environment-Aware Contrastive Cell Representation Learning for Histopathology](https://mlsanity.com/paper/2303.04696/) (2023)
- [Do ImageNet-trained models learn shortcuts? The impact of frequency shortcuts on generalization](https://mlsanity.com/paper/2503.03519/) (2025)
- [DFM-X: Augmentation by Leveraging Prior Knowledge of Shortcut Learning](https://mlsanity.com/paper/2308.06622/) (2023)
- [Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles](https://mlsanity.com/paper/2311.16176/) (2023)
- [Self-supervised learning for analysis of temporal and morphological drug effects in cancer cell imaging data](https://mlsanity.com/paper/2203.04289/) (2022)
- [Boosting Automatic COVID-19 Detection Performance with Self-Supervised Learning and Batch Knowledge Ensembling](https://mlsanity.com/paper/2212.09281/) (2022)
- [Harmonizing Pathological and Normal Pixels for Pseudo-healthy Synthesis](https://mlsanity.com/paper/2203.15347/) (2022)

## Abstract (arXiv)

In self-supervised visual representation learning, a feature extractor is trained on a "pretext task" for which labels can be generated cheaply, without human annotation. A central challenge in this approach is that the feature extractor quickly learns to exploit low-level visual features such as color aberrations or watermarks and then fails to learn useful semantic representations. Much work has gone into identifying such "shortcut" features and hand-designing schemes to reduce their effect. Here, we propose a general framework for mitigating the effect shortcut features. Our key assumption is that those features which are the first to be exploited for solving the pretext task may also be the most vulnerable to an adversary trained to make the task harder. We show that this assumption holds across common pretext tasks and datasets by training a "lens" network to make small image changes that maximally reduce performance in the pretext task. Representations learned with the modified images outperform those learned without in all tested cases. Additionally, the modifications made by the lens reveal how the choice of pretext task and dataset affects the features learned by self-supervision.