# Overcoming Shortcut Learning in a Target Domain by Generalizing Basic Visual Factors from a Source Domain

arXiv: https://arxiv.org/abs/2207.10002 · HTML: https://mlsanity.com/paper/2207.10002/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 20 Jul 2022.

## Builds on (most cited first, 22 of 22)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Learning to Generate Novel Domains for Domain Generalization](https://mlsanity.com/paper/2007.03304/) (2020)
- [Episodic Training for Domain Generalization](https://mlsanity.com/paper/1902.00113/) (2019)
- [Deep Domain-Adversarial Image Generation for Domain Generalisation](https://mlsanity.com/paper/2003.06054/) (2020)
- [Domain-Specific Batch Normalization for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1906.03950/) (2019)
- [Feature-Critic Networks for Heterogeneous Domain Generalization](https://mlsanity.com/paper/1901.11448/) (2019)
- [What shapes feature representations? Exploring datasets, architectures, and training](https://mlsanity.com/paper/2006.12433/) (2020)
- [Real-Time User-Guided Image Colorization with Learned Deep Priors](https://mlsanity.com/paper/1705.02999/) (2017)
- [Task-Driven Modular Networks for Zero-Shot Compositional Learning](https://mlsanity.com/paper/1905.05908/) (2019)
- [Learning Graph Embeddings for Compositional Zero-shot Learning](https://mlsanity.com/paper/2102.01987/) (2021)
- [Counterfactual Generative Networks](https://mlsanity.com/paper/2101.06046/) (2021)
- [Heterogeneous Domain Generalization via Domain Mixup](https://mlsanity.com/paper/2009.05448/) (2020)
- [Open World Compositional Zero-Shot Learning](https://mlsanity.com/paper/2101.12609/) (2021)
- [On Disentangled Representations Learned From Correlated Data](https://mlsanity.com/paper/2006.07886/) (2020)
- [A causal view of compositional zero-shot recognition](https://mlsanity.com/paper/2006.14610/) (2020)
- [Fruit recognition from images using deep learning](https://mlsanity.com/paper/1712.00580/) (2017)
- [DiagViB-6: A Diagnostic Benchmark Suite for Vision Models in the Presence of Shortcut and Generalization Opportunities](https://mlsanity.com/paper/2108.05779/) (2021)
- [Learning Single/Multi-Attribute of Object with Symmetry and Group](https://mlsanity.com/paper/2110.04603/) (2021)

## Built on by (most cited first, 7 of 7)

- [Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapse](https://mlsanity.com/paper/2405.05587/) (2024)
- [Why Do Probabilistic Clinical Models Fail To Transport Between Sites?](https://mlsanity.com/paper/2311.04787/) (2023)
- [When and How Does CLIP Enable Domain and Compositional Generalization?](https://mlsanity.com/paper/2502.09507/) (2025)
- [DFM-X: Augmentation by Leveraging Prior Knowledge of Shortcut Learning](https://mlsanity.com/paper/2308.06622/) (2023)
- [Revisiting Adaptive Cellular Recognition Under Domain Shifts: A Contextual Correspondence View](https://mlsanity.com/paper/2407.12870/) (2024)
- [AutoMatch: A Large-scale Audio Beat Matching Benchmark for Boosting Deep Learning Assistant Video Editing](https://mlsanity.com/paper/2303.01884/) (2023)
- [Benign Shortcut for Debiasing: Fair Visual Recognition via Intervention with Shortcut Features](https://mlsanity.com/paper/2308.08482/) (2023)

## Abstract (arXiv)

Shortcut learning occurs when a deep neural network overly relies on spurious correlations in the training dataset in order to solve downstream tasks. Prior works have shown how this impairs the compositional generalization capability of deep learning models. To address this problem, we propose a novel approach to mitigate shortcut learning in uncontrolled target domains. Our approach extends the training set with an additional dataset (the source domain), which is specifically designed to facilitate learning independent representations of basic visual factors. We benchmark our idea on synthetic target domains where we explicitly control shortcut opportunities as well as real-world target domains. Furthermore, we analyze the effect of different specifications of the source domain and the network architecture on compositional generalization. Our main finding is that leveraging data from a source domain is an effective way to mitigate shortcut learning. By promoting independence across different factors of variation in the learned representations, networks can learn to consider only predictive factors and ignore potential shortcut factors during inference.