mlsanityData to 7 Oct 2026

On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learning

arXiv 2406.10815 ↗First posted 16 Jun 2024cs.LG cs.AI cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Supervised contrastive representation learning has been shown to be effective in various transfer learning scenarios. However, while asymmetric non-contrastive learning (ANCL) often outperforms its contrastive learning counterpart in self-supervised representation learning, the extension of ANCL to supervised scenarios is less explored. To bridge the gap, we study ANCL for supervised representation learning, coined SupSiam and SupBYOL, leveraging labels in ANCL to achieve better representations. The proposed supervised ANCL framework improves representation learning while avoiding collapse. Our analysis reveals that providing supervision to ANCL reduces intra-class variance, and the contribution of supervision should be adjusted to achieve the best performance. Experiments demonstrate the superiority of supervised ANCL across various datasets and tasks. The code is available at: https://github.com/JH-Oh-23/Sup-ANCL.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 38 of its 63 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

38 of 63 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,853
26 more
  1. 2020Supervised Contrastive Learningcited by 2,803
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  11. 2018Do Better ImageNet Models Transfer Better?cited by 560
  12. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  13. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  14. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  15. 2021Co2L: Contrastive Continual Learningcited by 243
  16. 2021Improving Transferability of Representations via Augmentation-Aware Self-Supervisioncited by 48
  17. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  18. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
  19. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
  20. 2020Can Semantic Labels Assist Self-Supervised Visual Representation Learning?cited by 16
  21. 2022Implicit variance regularization in non-contrastive SSLcited by 15
  22. 2023The Edge of Orthogonality: A Simple View of What Makes BYOL Tickcited by 15
  23. 2021Supervised Momentum Contrastive Learning for Few-Shot Classificationcited by 6
  24. 2022Tackling Online One-Class Incremental Learning by Removing Negative Contrastscited by 4
  25. 2022OPERA: Omni-Supervised Representation Learning with Hierarchical Supervisionscited by 3
  26. 2023SupSiam: Non-contrastive Auxiliary Loss for Learning from Molecular Conformerscited by 3

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Modeling Hierarchical Spaces: A Review and Unified Framework for Surrogate-Based Architecture Designcited by 3
  2. 2025A Theoretical Framework for Preventing Class Collapse in Supervised Contrastive Learningcited by 2
  3. 2025Soft Task-Aware Routing of Experts for Equivariant Representation Learningcited by 0