mlsanityData to 7 Oct 2026

Exploring Simple Siamese Representation Learning

arXiv 2011.10566 ↗First posted 20 Nov 2020cs.CV cs.LGcited by 2,853

Not on the map. Why?

Cited by 61 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code will be made available.

Key facts. Cited by 2,853 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 365; 2022: 700; 2023: 691; 2024: 508; 2025: 359; 2026: 230.

2,853 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  9. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  10. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  11. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  12. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
38 more
  1. 2021Extract Free Dense Labels from CLIPcited by 580
  2. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  3. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  4. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  5. 2021Image Segmentation Using Text and Image Promptscited by 509
  6. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  7. 2023Emergent Correspondence from Image Diffusioncited by 437
  8. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  9. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 402
  10. 2022SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentationcited by 397
  11. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  12. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  13. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  14. 2021Time-Series Representation Learning via Temporal and Contextual Contrastingcited by 342
  15. 2021Transformers Can Do Bayesian Inferencecited by 338
  16. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  17. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  18. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  19. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  20. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  21. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  22. 2022Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learningcited by 302
  23. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  24. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  25. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  26. 2021Graph Self-Supervised Learning: A Surveycited by 289
  27. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  28. 2023Towards Unbounded Machine Unlearningcited by 261
  29. 2022Contrastive Test-Time Adaptationcited by 258
  30. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  31. 2023SpectralGPT: Spectral Remote Sensing Foundation Modelcited by 250
  32. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  33. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  34. 2023A Cookbook of Self-Supervised Learningcited by 248
  35. 2021Co2L: Contrastive Continual Learningcited by 243
  36. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  37. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  38. 2021Mastering Atari Games with Limited Datacited by 235

50 most cited of 2,853 All 2,853 (JSON)