mlsanityData to 14 Oct 2026

Understanding self-supervised Learning Dynamics without Contrastive Pairs

arXiv 2102.06810 ↗First posted 12 Feb 2021cs.LG cs.AI cs.CVcited by 256

Not on the map. Why?

Abstract (arXiv)

While contrastive approaches of self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing views from different data points (negative pairs), recent non-contrastive SSL (e.g., BYOL and SimSiam) show remarkable performance it without negative pairs, with an extra learnable predictor and a stop-gradient operation. A fundamental question arises: why do these methods not collapse into trivial representations? We answer this question via a simple theoretical study and propose a novel approach, DirectPred, that directly sets the linear predictor based on the statistics of its inputs, without gradient training. On ImageNet, it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm and outperforms a linear predictor by 2.5% in 300-epoch training (and 5% in 60-epoch). DirectPred is motivated by our theoretical study of the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our study yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Code is released https://github.com/facebookresearch/luckmatters/tree/master/ssl.

Key facts. Cited by 256 papers in the map (data through 14 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

30 of 36 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  9. 2019Contrastive Multiview Codingcited by 1,301
  10. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  11. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  12. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
18 more
  1. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  2. 2017Large Batch Training of Convolutional Networkscited by 535
  3. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  4. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  5. 2016Deep Learning without Poor Local Minimacited by 234
  6. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  7. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  8. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  9. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  10. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  11. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  12. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  13. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  14. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 65
  15. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  16. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  17. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  18. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 35; 2022: 78; 2023: 52; 2024: 38; 2025: 25; 2026: 28.

256 papers.

Sort:

By area:

  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  2. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  3. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  4. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  5. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  6. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  7. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  8. 2023A Cookbook of Self-Supervised Learningcited by 250
  9. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  10. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  11. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  12. 2021Self-Supervised Models are Continual Learnerscited by 149
38 more
  1. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  2. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 141
  3. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  4. 2021Targeted Supervised Contrastive Learning for Long-Tailed Recognitioncited by 132
  5. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 123
  6. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  7. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  8. 2021Large-scale Unsupervised Semantic Segmentationcited by 101
  9. 2023What Do Self-Supervised Vision Transformers Learn?cited by 93
  10. 2022Divergence-aware Federated Self-Supervised Learningcited by 73
  11. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  12. 2022Crafting Better Contrastive Views for Siamese Representation Learningcited by 67
  13. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  14. 2022Audio Self-supervised Learning: A Surveycited by 65
  15. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  16. 2022Video Transformers: A Surveycited by 63
  17. 2021Learning Representation for Clustering via Prototype Scattering and Positive Samplingcited by 58
  18. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 58
  19. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  20. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  21. 2022Semi-Supervised and Unsupervised Deep Visual Learning: A Surveycited by 52
  22. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  23. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  24. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  25. 2021Self-Supervised Learning Disentangled Group Representation as Featurecited by 47
  26. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  27. 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 44
  28. 2023Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspectivecited by 43
  29. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  30. 2022Understanding Collapse in Non-Contrastive Siamese Representation Learningcited by 42
  31. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  32. 2022On the Importance of Asymmetry for Siamese Representation Learningcited by 39
  33. 2022Do More Negative Samples Necessarily Hurt in Contrastive Learning?cited by 39
  34. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  35. 2021Continually Learning Self-Supervised Representations with Projected Functional Regularizationcited by 38
  36. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  37. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  38. 2021Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspectivecited by 37

50 most cited of 256 All 256 (JSON)