# How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning

arXiv: https://arxiv.org/abs/2203.16262 · HTML: https://mlsanity.com/paper/2203.16262/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [Blind Super-Resolution Kernel Estimation using an Internal-GAN](https://mlsanity.com/paper/1909.06581/) (2019)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)
- [BYOL works even without batch statistics](https://mlsanity.com/paper/2010.10241/) (2020)
- [DC-BERT: Decoupling Question and Document for Efficient Contextual Encoding](https://mlsanity.com/paper/2002.12591/) (2020)

## Built on by (most cited first, 30 of 65)

- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection](https://mlsanity.com/paper/2306.10347/) (2023)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [ReContrast: Domain-Specific Anomaly Detection via Contrastive Reconstruction](https://mlsanity.com/paper/2306.02602/) (2023)
- [A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond](https://mlsanity.com/paper/2208.00173/) (2022)
- [One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era](https://mlsanity.com/paper/2304.06488/) (2023)
- [Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing](https://mlsanity.com/paper/2401.15855/) (2024)
- [A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering](https://mlsanity.com/paper/2306.06211/) (2023)
- [AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders](https://mlsanity.com/paper/2211.09120/) (2022)
- [The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning](https://mlsanity.com/paper/2205.06226/) (2022)
- [Improving Self-Supervised Learning by Characterizing Idealized Representations](https://mlsanity.com/paper/2209.06235/) (2022)
- [Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism](https://mlsanity.com/paper/2303.02387/) (2023)
- [An Investigation into Whitening Loss for Self-supervised Learning](https://mlsanity.com/paper/2210.03586/) (2022)
- [Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness](https://mlsanity.com/paper/2207.10899/) (2022)
- [MP-FedCL: Multiprototype Federated Contrastive Learning for Edge Intelligence](https://mlsanity.com/paper/2304.01950/) (2023)
- [Implicit variance regularization in non-contrastive SSL](https://mlsanity.com/paper/2212.04858/) (2022)
- [Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approach](https://mlsanity.com/paper/2302.03357/) (2023)
- [PatchAD: A Lightweight Patch-based MLP-Mixer for Time Series Anomaly Detection](https://mlsanity.com/paper/2401.09793/) (2024)
- [Cross-view Masked Diffusion Transformers for Person Image Synthesis](https://mlsanity.com/paper/2402.01516/) (2024)
- [On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2208.05744/) (2022)
- [A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learning](https://mlsanity.com/paper/2303.12710/) (2023)
- [TimeDRL: Disentangled Representation Learning for Multivariate Time-Series](https://mlsanity.com/paper/2312.04142/) (2023)
- [FreCT: Frequency-augmented Convolutional Transformer for Robust Time Series Anomaly Detection](https://mlsanity.com/paper/2505.00941/) (2025)
- [Modulate Your Spectrum in Self-Supervised Learning](https://mlsanity.com/paper/2305.16789/) (2023)
- [On the Importance of Embedding Norms in Self-Supervised Learning](https://mlsanity.com/paper/2502.09252/) (2025)
- [The Common Stability Mechanism behind most Self-Supervised Learning Approaches](https://mlsanity.com/paper/2402.14957/) (2024)
- [Self-Supervised Visual Representation Learning via Residual Momentum](https://mlsanity.com/paper/2211.09861/) (2022)
- [Simple Contrastive Representation Learning for Time Series Forecasting](https://mlsanity.com/paper/2303.18205/) (2023)
- [Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off](https://mlsanity.com/paper/2402.14648/) (2024)

## Abstract (arXiv)

To avoid collapse in self-supervised learning (SSL), a contrastive loss is widely used but often requires a large number of negative samples. Without negative samples yet achieving competitive performance, a recent work has attracted significant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse. However, the reason for how it avoids collapse without negative samples remains not fully clear and our investigation starts by revisiting the explanatory claims in the original SimSiam. After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of the l2-normalized representation vector. This yields a unified perspective on how negative samples and SimSiam alleviate collapse. Such a unified perspective comes timely for understanding the recent progress in SSL.