# Understanding Collapse in Non-Contrastive Siamese Representation Learning

arXiv: https://arxiv.org/abs/2209.15007 · HTML: https://mlsanity.com/paper/2209.15007/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 29 Sep 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases](https://mlsanity.com/paper/2007.13916/) (2020)
- [BYOL works even without batch statistics](https://mlsanity.com/paper/2010.10241/) (2020)
- [Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning](https://mlsanity.com/paper/2110.04593/) (2021)
- [Online Continual Learning For Visual Food Classification](https://mlsanity.com/paper/2108.06781/) (2021)
- [The Challenges of Continuous Self-Supervised Learning](https://mlsanity.com/paper/2203.12710/) (2022)

## Built on by (most cited first, 30 of 42)

- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank](https://mlsanity.com/paper/2210.02885/) (2022)
- [EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars](https://mlsanity.com/paper/2404.19110/) (2024)
- [CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding](https://mlsanity.com/paper/2304.09670/) (2023)
- [A General-Purpose Self-Supervised Model for Computational Pathology](https://mlsanity.com/paper/2308.15474/) (2023)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures](https://mlsanity.com/paper/2312.04000/) (2023)
- [HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions](https://mlsanity.com/paper/2304.00387/) (2023)
- [Internet Explorer: Targeted Representation Learning on the Open Web](https://mlsanity.com/paper/2302.14051/) (2023)
- [VISReg: Variance-Invariance-Sketching Regularization for JEPA training](https://mlsanity.com/paper/2606.02572/) (2026)
- [Video Representation Learning with Joint-Embedding Predictive Architectures](https://mlsanity.com/paper/2412.10925/) (2024)
- [Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data](https://mlsanity.com/paper/2403.16398/) (2024)
- [Self-Supervised Representation Learning with Joint Embedding Predictive Architecture for Automotive LiDAR Object Detection](https://mlsanity.com/paper/2501.04969/) (2025)
- [LDReg: Local Dimensionality Regularized Self-Supervised Learning](https://mlsanity.com/paper/2401.10474/) (2024)
- [Contrastive Learning with Consistent Representations](https://mlsanity.com/paper/2302.01541/) (2023)
- [Deciphering the Projection Head: Representation Evaluation Self-supervised Learning](https://mlsanity.com/paper/2301.12189/) (2023)
- [You Are Your Own Best Teacher: Achieving Centralized-level Performance in Federated Learning under Heterogeneous and Long-tailed Data](https://mlsanity.com/paper/2503.06916/) (2025)
- [LayerLock: Non-collapsing Representation Learning with Progressive Freezing](https://mlsanity.com/paper/2509.10156/) (2025)
- [SupSiam: Non-contrastive Auxiliary Loss for Learning from Molecular Conformers](https://mlsanity.com/paper/2302.07754/) (2023)
- [SimSMoE: Solving Representational Collapse via Similarity Measure](https://mlsanity.com/paper/2406.15883/) (2024)
- [Enhancing Out-of-Distribution Detection with Extended Logit Normalization](https://mlsanity.com/paper/2504.11434/) (2025)
- [Can representation learning for multimodal image registration be improved by supervision of intermediate layers?](https://mlsanity.com/paper/2303.00403/) (2023)
- [Self-Supervised Visual Representation Learning on Food Images](https://mlsanity.com/paper/2303.09046/) (2023)
- [Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamics](https://mlsanity.com/paper/2309.16109/) (2023)
- [On Learning Discriminative Features from Synthesized Data for Self-Supervised Fine-Grained Visual Recognition](https://mlsanity.com/paper/2407.14676/) (2024)
- [T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2510.23484/) (2025)
- [Global Geometry Is Not Enough for Vision Representations](https://mlsanity.com/paper/2602.03282/) (2026)
- [Self-Supervised Learning as Discrete Communication](https://mlsanity.com/paper/2602.09764/) (2026)
- [Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defense](https://mlsanity.com/paper/2409.08509/) (2024)
- [PP-SSL : Priority-Perception Self-Supervised Learning for Fine-Grained Recognition](https://mlsanity.com/paper/2412.00134/) (2024)

## Abstract (arXiv)

Contrastive methods have led a recent surge in the performance of self-supervised representation learning (SSL). Recent methods like BYOL or SimSiam purportedly distill these contrastive methods down to their essence, removing bells and whistles, including the negative examples, that do not contribute to downstream performance. These "non-contrastive" methods work surprisingly well without using negatives even though the global minimum lies at trivial collapse. We empirically analyze these non-contrastive methods and find that SimSiam is extraordinarily sensitive to dataset and model size. In particular, SimSiam representations undergo partial dimensional collapse if the model is too small relative to the dataset size. We propose a metric to measure the degree of this collapse and show that it can be used to forecast the downstream task performance without any fine-tuning or labels. We further analyze architectural design choices and their effect on the downstream performance. Finally, we demonstrate that shifting to a continual learning setting acts as a regularizer and prevents collapse, and a hybrid between continual and multi-epoch training can improve linear probe accuracy by as many as 18 percentage points using ResNet-18 on ImageNet. Our project page is at https://alexanderli.com/noncontrastive-ssl/.