# How Robust is Unsupervised Representation Learning to Distribution Shift?

arXiv: https://arxiv.org/abs/2206.08871 · HTML: https://mlsanity.com/paper/2206.08871/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 55 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Builds on (most cited first, 30 of 43)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Gradient Matching for Domain Generalization](https://mlsanity.com/paper/2104.09937/) (2021)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [SGD on Neural Networks Learns Functions of Increasing Complexity](https://mlsanity.com/paper/1905.11604/) (2019)
- [Adversarial Filters of Dataset Biases](https://mlsanity.com/paper/2002.04108/) (2020)

## Built on by (most cited first, 24 of 24)

- [Group Robust Classification Without Any Group Information](https://mlsanity.com/paper/2310.18555/) (2023)
- [Learning Graph Quantized Tokenizers](https://mlsanity.com/paper/2410.13798/) (2024)
- [FRAPPE: A Group Fairness Framework for Post-Processing Everything](https://mlsanity.com/paper/2312.02592/) (2023)
- [Intermediate Layer Classifiers for OOD generalization](https://mlsanity.com/paper/2504.05461/) (2025)
- [Does Progress On Object Recognition Benchmarks Improve Real-World Generalization?](https://mlsanity.com/paper/2307.13136/) (2023)
- [Continual Learning for Predictive Maintenance: Overview and Challenges](https://mlsanity.com/paper/2301.12467/) (2023)
- [Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation](https://mlsanity.com/paper/2406.19049/) (2024)
- [The Role of Learning Algorithms in Collective Action](https://mlsanity.com/paper/2405.06582/) (2024)
- [On Pretraining Data Diversity for Self-Supervised Learning](https://mlsanity.com/paper/2403.13808/) (2024)
- [Unsupervised Camouflaged Object Segmentation as Domain Adaptation](https://mlsanity.com/paper/2308.04528/) (2023)
- [OODRobustBench: a Benchmark and Large-Scale Analysis of Adversarial Robustness under Distribution Shift](https://mlsanity.com/paper/2310.12793/) (2023)
- [Identifying and Disentangling Spurious Features in Pretrained Image Representations](https://mlsanity.com/paper/2306.12673/) (2023)
- [In-context Learning in Presence of Spurious Correlations](https://mlsanity.com/paper/2410.03140/) (2024)
- [Towards Brain MRI Foundation Models for the Clinic: Findings from the FOMO25 Challenge](https://mlsanity.com/paper/2604.11679/) (2026)
- [Robustness May be More Brittle than We Think under Different Degrees of Distribution Shifts](https://mlsanity.com/paper/2310.06622/) (2023)
- [On the Adversarial Robustness of Graph Contrastive Learning Methods](https://mlsanity.com/paper/2311.17853/) (2023)
- [A Foundation Model for Wearable Movement Data in Mental Health Research](https://mlsanity.com/paper/2411.15240/) (2024)
- [SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval](https://mlsanity.com/paper/2505.15867/) (2025)
- [CTA: Cross-Task Alignment for Better Test Time Training](https://mlsanity.com/paper/2507.05221/) (2025)
- [Unsupervised energy disaggregation via convolutional sparse coding](https://mlsanity.com/paper/2207.09785/) (2022)
- [A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attention](https://mlsanity.com/paper/2407.18673/) (2024)
- [Trend-Aware Supervision: On Learning Invariance for Semi-Supervised Facial Action Unit Intensity Estimation](https://mlsanity.com/paper/2503.08078/) (2025)
- [DynaSubVAE: Adaptive Subgrouping for Scalable and Robust OOD Detection](https://mlsanity.com/paper/2506.10200/) (2025)
- [Active Learning with Task-Driven Representations for Messy Pools](https://mlsanity.com/paper/2510.25926/) (2025)

## Abstract (arXiv)

The robustness of machine learning algorithms to distributions shift is primarily discussed in the context of supervised learning (SL). As such, there is a lack of insight on the robustness of the representations learned from unsupervised methods, such as self-supervised learning (SSL) and auto-encoder based algorithms (AE), to distribution shift. We posit that the input-driven objectives of unsupervised algorithms lead to representations that are more robust to distribution shift than the target-driven objective of SL. We verify this by extensively evaluating the performance of SSL and AE on both synthetic and realistic distribution shift datasets. Following observations that the linear layer used for classification itself can be susceptible to spurious correlations, we evaluate the representations using a linear head trained on a small amount of out-of-distribution (OOD) data, to isolate the robustness of the learned representations from that of the linear head. We also develop "controllable" versions of existing realistic domain generalisation datasets with adjustable degrees of distribution shifts. This allows us to study the robustness of different learning algorithms under versatile yet realistic distribution shift conditions. Our experiments show that representations learned from unsupervised learning algorithms generalise better than SL under a wide variety of extreme as well as realistic distribution shifts.