mlsanityData to 7 Oct 2026

On the Importance of Asymmetry for Siamese Representation Learning

arXiv 2204.00613 ↗First posted 1 Apr 2022cs.CV cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Many recent self-supervised frameworks for visual representation learning are based on certain forms of Siamese networks. Such networks are conceptually symmetric with two parallel encoders, but often practically asymmetric as numerous mechanisms are devised to break the symmetry. In this work, we conduct a formal study on the importance of asymmetry by explicitly distinguishing the two encoders within the network -- one produces source encodings and the other targets. Our key insight is keeping a relatively lower variance in target than source generally benefits learning. This is empirically justified by our results from five case studies covering different variance-oriented designs, and is aligned with our preliminary theoretical analysis on the baseline. Moreover, we find the improvements from asymmetric designs generalize well to longer training schedules, multiple other frameworks and newer backbones. Finally, the combined effect of several asymmetric designs achieves a state-of-the-art accuracy on ImageNet linear probing and competitive results on downstream transfer. We hope our exploration will inspire more research in exploiting asymmetry for Siamese representation learning.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 40 of its 47 references that are in the map. First posted on arXiv 1 Apr 2022.

Builds on

40 of 47 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,853
28 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  4. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2018Group Normalizationcited by 1,514
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  12. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  13. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  14. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  15. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
  16. 2017Large Batch Training of Convolutional Networkscited by 529
  17. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  18. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  19. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  20. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  21. 2021Decoupled Contrastive Learningcited by 141
  22. 2018Iterative Learning with Open-set Noisy Labelscited by 108
  23. 2021Contrastive Learning with Stronger Augmentationscited by 100
  24. 2021Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learningcited by 78
  25. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  26. 2021Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learningcited by 14
  27. 2019EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformationscited by 11
  28. 2021A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 15; 2024: 8; 2025: 5; 2026: 1.

38 papers.

Sort:

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  3. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  4. 2022Masked Image Modeling with Denoising Contrastcited by 45
  5. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
  6. 2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 30
  7. 2022No Reason for No Supervision: Improved Generalization in Supervised Modelscited by 23
  8. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  9. 2022Masked Siamese ConvNetscited by 22
  10. 2023PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videoscited by 22
  11. 2022Weighted Ensemble Self-Supervised Learningcited by 19
  12. 2023Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videoscited by 16
26 more
  1. 2022Implicit variance regularization in non-contrastive SSLcited by 15
  2. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
  3. 2024Augmentations vs Algorithms: What Works in Self-Supervised Learningcited by 9
  4. 2024Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networkscited by 9
  5. 2022CropMix: Sampling a Rich Input Distribution via Multi-Scale Croppingcited by 8
  6. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  7. 2024Understanding Self-Supervised Learning via Gaussian Mixture Modelscited by 6
  8. 2022Masked Contrastive Representation Learningcited by 5
  9. 2023Asymmetric Patch Sampling for Contrastive Learningcited by 5
  10. 2023Masking meets Supervision: A Strong Learning Alliancecited by 3
  11. 2023The Triad of Failure Modes and a Possible Way Outcited by 3
  12. 2025LayerLock: Non-collapsing Representation Learning with Progressive Freezingcited by 3
  13. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
  14. 2023Asymmetric Masked Distillation for Pre-Training Small Foundation Modelscited by 2
  15. 2023Contrastive learning-based agent modeling for deep reinforcement learningcited by 2
  16. 2023Unifying Self-Supervised Clustering and Energy-Based Modelscited by 2
  17. 2024Cohere3D: Exploiting Temporal Coherence for Unsupervised Representation Learning of Vision-based Autonomous Drivingcited by 2
  18. 2024Temporal Reversal Regularization for Spiking Neural Networks: Hybrid Spatio-Temporal Invariance for Generalizationcited by 2
  19. 2023ScoreCL: Augmentation-Adaptive Contrastive Learning via Score-Matching Functioncited by 1
  20. 2023Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcitycited by 1
  21. 2023Masking Improves Contrastive Self-Supervised Learning for ConvNets, and Saliency Tells You Wherecited by 1
  22. 2024Extract More from Less: Efficient Fine-Grained Visual Recognition in Low-Data Regimescited by 1
  23. 2025Implicit Contrastive Representation Learning with Guided Stop-gradientcited by 1
  24. 2025Multimodal 3D Genome Pre-trainingcited by 0
  25. 2025Understanding Self-supervised Contrastive Learning through Supervised Objectivescited by 0
  26. 2026Self-Supervised Learning with a Multi-Task Latent Space Objectivecited by 0