mlsanityData to 14 Oct 2026

Towards Demystifying Representation Learning with Non-contrastive Self-supervision

arXiv 2110.04947 ↗First posted 11 Oct 2021cs.LG stat.MLcited by 23

Not on the map. Why?

Abstract (arXiv)

Non-contrastive methods of self-supervised learning (such as BYOL and SimSiam) learn representations by minimizing the distance between two views of the same image. These approaches have achieved remarkable performance in practice, but the theoretical understanding lags behind. Tian et al. 2021 explained why the representation does not collapse to zero, however, how the feature is learned still remains mysterious. In our work, we prove in a linear network, non-contrastive methods learn a desirable projection matrix and also reduce the sample complexity on downstream tasks. Our analysis suggests that weight decay acts as an implicit threshold that discards the features with high variance under data augmentations, and keeps the features with low variance. Inspired by our theory, we design a simpler and more computationally efficient algorithm DirectCopy by removing the eigen-decomposition step in the original DirectPred algorithm in Tian et al. 2021. Our experiments show that DirectCopy rivals or even outperforms DirectPred on STL-10, CIFAR-10, CIFAR-100, and ImageNet.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 11 Oct 2021.

Builds on

32 of 42 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  11. 2019Contrastive Multiview Codingcited by 1,301
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
20 more
  1. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  3. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  4. 2017Large Batch Training of Convolutional Networkscited by 535
  5. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  6. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  7. 2020Whitening for Self-Supervised Representation Learningcited by 279
  8. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  9. 2016Deep Learning without Poor Local Minimacited by 234
  10. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  11. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  12. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  13. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  14. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  15. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  16. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  17. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 65
  18. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  19. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  20. 2015Learning Mixtures of Gaussians in High Dimensionscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 7; 2024: 3; 2025: 4; 2026: 2.

23 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 250
  2. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  3. 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 44
  4. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  5. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 26
  6. 2022Contrasting the landscape of contrastive and non-contrastive learningcited by 24
  7. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20
  8. 2022Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learningcited by 19
  9. 2022Implicit variance regularization in non-contrastive SSLcited by 16
  10. 2023The Edge of Orthogonality: A Simple View of What Makes BYOL Tickcited by 15
  11. 2023SemPPL: Predicting pseudo-labels for better contrastive representationscited by 6
  12. 2023Random Teachers are Good Teacherscited by 6
11 more
  1. 2025Representation Learning via Non-Contrastive Mutual Informationcited by 5
  2. 2023Rethinking Evaluation Protocols of Visual Representations Learned via Self-supervised Learningcited by 4
  3. 2025Dual Perspectives on Non-Contrastive Self-Supervised Learningcited by 4
  4. 2025Clustering via Self-Supervised Diffusioncited by 3
  5. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
  6. 2022One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learningcited by 1
  7. 2024Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defensecited by 1
  8. 2023A Spatiotemporal Correspondence Approach to Unsupervised LiDAR Segmentation with Traffic Applicationscited by 0
  9. 2025Self-Supervised Representation Learning as Mutual Information Maximizationcited by 0
  10. 2026Spectral Ghost in Representation Learning: from Component Analysis to Self-Supervised Learningcited by 0
  11. 2026Randomly Initialized Networks Can Learn from Peer-to-Peer Consensuscited by 0