mlsanityData to 7 Oct 2026

How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning

arXiv 2203.16262 ↗First posted 30 Mar 2022cs.LG cs.AIcited by 65

Not on the map. Why?

Abstract (arXiv)

To avoid collapse in self-supervised learning (SSL), a contrastive loss is widely used but often requires a large number of negative samples. Without negative samples yet achieving competitive performance, a recent work has attracted significant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse. However, the reason for how it avoids collapse without negative samples remains not fully clear and our investigation starts by revisiting the explanatory claims in the original SimSiam. After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of the l2-normalized representation vector. This yields a unified perspective on how negative samples and SimSiam alleviate collapse. Such a unified perspective comes timely for understanding the recent progress in SSL.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 28 of its 34 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

28 of 34 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  10. 2020Supervised Contrastive Learningcited by 2,803
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
16 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  4. 2019Contrastive Multiview Codingcited by 1,297
  5. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  8. 2020Understanding the Behaviour of Contrastive Losscited by 397
  9. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  10. 2020Whitening for Self-Supervised Representation Learningcited by 278
  11. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  12. 2019Blind Super-Resolution Kernel Estimation using an Internal-GANcited by 175
  13. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  14. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  15. 2020BYOL works even without batch statisticscited by 95
  16. 2020DC-BERT: Decoupling Question and Document for Efficient Contextual Encodingcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 21; 2024: 18; 2025: 12; 2026: 3.

65 papers.

Sort:

By area:

  1. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 345
  2. 2023A Cookbook of Self-Supervised Learningcited by 248
  3. 2023DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detectioncited by 103
  4. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  5. 2023ReContrast: Domain-Specific Anomaly Detection via Contrastive Reconstructioncited by 62
  6. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  7. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  8. 2024Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensingcited by 47
  9. 2023A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineeringcited by 45
  10. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 41
  11. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  12. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
38 more
  1. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
  2. 2022An Investigation into Whitening Loss for Self-supervised Learningcited by 21
  3. 2022Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustnesscited by 20
  4. 2023MP-FedCL: Multiprototype Federated Contrastive Learning for Edge Intelligencecited by 20
  5. 2022Implicit variance regularization in non-contrastive SSLcited by 15
  6. 2023Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approachcited by 13
  7. 2024PatchAD: A Lightweight Patch-based MLP-Mixer for Time Series Anomaly Detectioncited by 12
  8. 2024Cross-view Masked Diffusion Transformers for Person Image Synthesiscited by 12
  9. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
  10. 2023A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learningcited by 10
  11. 2023TimeDRL: Disentangled Representation Learning for Multivariate Time-Seriescited by 10
  12. 2025FreCT: Frequency-augmented Convolutional Transformer for Robust Time Series Anomaly Detectioncited by 10
  13. 2023Modulate Your Spectrum in Self-Supervised Learningcited by 9
  14. 2025On the Importance of Embedding Norms in Self-Supervised Learningcited by 9
  15. 2024The Common Stability Mechanism behind most Self-Supervised Learning Approachescited by 8
  16. 2022Self-Supervised Visual Representation Learning via Residual Momentumcited by 7
  17. 2023Simple Contrastive Representation Learning for Time Series Forecastingcited by 7
  18. 2024The Hidden Pitfalls of the Cosine Similarity Losscited by 7
  19. 2024ToCoAD: Two-Stage Contrastive Learning for Industrial Anomaly Detectioncited by 7
  20. 2023Random Teachers are Good Teacherscited by 6
  21. 2024Analysis of Using Sigmoid Loss for Contrastive Learningcited by 6
  22. 2024Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Lossescited by 6
  23. 2025DeRA: Decoupled Representation Alignment for Video Tokenizationcited by 6
  24. 2023BSL: Understanding and Improving Softmax Loss for Recommendationcited by 5
  25. 2024Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-offcited by 5
  26. 2024FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Datacited by 5
  27. 2024Lite-SAM Is Actually What You Need for Segment Everythingcited by 5
  28. 2023Towards the Sparseness of Projection Head in Self-Supervised Learningcited by 4
  29. 2024DeepSet SimCLR: Self-supervised deep sets for improved pathology representation learningcited by 4
  30. 2024Rethinking The Uniformity Metric in Self-Supervised Learningcited by 3
  31. 2025Dual Perspectives on Non-Contrastive Self-Supervised Learningcited by 3
  32. 2025ZeroSiam: An Efficient Asymmetry for Test-Time Entropy Optimization without Collapsecited by 3
  33. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
  34. 2024Self-Supervised Learning of Disentangled Representations for Multivariate Time-Seriescited by 2
  35. 2025E-MD3C: Taming Masked Diffusion Transformers for Efficient Zero-Shot Object Customizationcited by 2
  36. 2026A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformerscited by 2
  37. 2022Self-Supervised Representation Learning With MUlti-Segmental Informational Coding (MUSIC)cited by 1
  38. 2023Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcitycited by 1

50 most cited of 65 All 65 (JSON)