mlsanityData to 14 Oct 2026

The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning

arXiv 2205.06226 ↗First posted 12 May 2022cs.LG stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

Recently the surprising discovery of the Bootstrap Your Own Latent (BYOL) method by Grill et al. shows the negative term in contrastive loss can be removed if we add the so-called prediction head to the network. This initiated the research of non-contrastive self-supervised learning. It is mysterious why even when there exist trivial collapsed global optimal solutions, neural networks trained by (stochastic) gradient descent can still learn competitive representations. This phenomenon is a typical example of implicit bias in deep learning and remains little understood. In this work, we present our empirical and theoretical discoveries on non-contrastive self-supervised learning. Empirically, we find that when the prediction head is initialized as an identity matrix with only its off-diagonal entries being trainable, the network can learn competitive representations even though the trivial optima still exist in the training objective. Theoretically, we present a framework to understand the behavior of the trainable, but identity-initialized prediction head. Under a simple setting, we characterized the substitution effect and acceleration effect of the prediction head. The substitution effect happens when learning the stronger features in some neurons can substitute for learning these features in other neurons through updating the prediction head. And the acceleration effect happens when the substituted features can accelerate the learning of other weaker features to prevent them from being ignored. These two effects enable the neural networks to learn all the features rather than focus only on learning the stronger features, which is likely the cause of the dimensional collapse phenomenon. To the best of our knowledge, this is also the first end-to-end optimization guarantee for non-contrastive methods using nonlinear neural networks with a trainable prediction head and normalization.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 76 of its 91 references that are in the map. First posted on arXiv 12 May 2022.

Builds on

76 of 91 references.

References · 76

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,866
38 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  8. 2019Contrastive Multiview Codingcited by 1,301
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  10. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  11. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  12. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  13. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  14. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  15. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  16. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  17. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  18. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  19. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  20. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  21. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  22. 2020Whitening for Self-Supervised Representation Learningcited by 279
  23. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  24. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  25. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  26. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  27. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  28. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  29. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  30. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  31. 2020Intriguing Properties of Contrastive Lossescited by 130
  32. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  33. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  34. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  35. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  36. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  37. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 102
  38. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 13; 2024: 9; 2025: 7; 2026: 2.

38 papers.

Sort:

By area:

  1. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  2. 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 44
  3. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  4. 2022On the Power of Foundation Modelscited by 37
  5. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  6. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 36
  7. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 33
  8. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  9. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  10. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20
  11. 2023Information Flow in Self-Supervised Learningcited by 16
  12. 2024What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encodingcited by 14
26 more
  1. 2024Identifiable Causal Representation Learning: Unsupervised, Multi-View, and Multi-Environmentcited by 12
  2. 2022Link Prediction with Non-Contrastive Learningcited by 11
  3. 2023More Synergy, Less Redundancy: Exploiting Joint Mutual Information for Self-Supervised Learningcited by 11
  4. 2025Projection Head is Secretly an Information Bottleneckcited by 11
  5. 2024iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learningcited by 9
  6. 2024Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-offcited by 7
  7. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  8. 2022GEDI: GEnerative and DIscriminative Training for Self-Supervised Learningcited by 6
  9. 2023Unraveling Projection Heads in Contrastive Learning: Insights from Expansion and Shrinkagecited by 6
  10. 2024Understanding Self-Supervised Learning via Gaussian Mixture Modelscited by 6
  11. 2023Deciphering the Projection Head: Representation Evaluation Self-supervised Learningcited by 5
  12. 2024Towards Understanding Why FixMatch Generalizes Better Than Supervised Learningcited by 5
  13. 2023Towards the Sparseness of Projection Head in Self-Supervised Learningcited by 4
  14. 2025Dual Perspectives on Non-Contrastive Self-Supervised Learningcited by 4
  15. 2022Augmentation Invariant Manifold Learningcited by 2
  16. 2023Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learningcited by 2
  17. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
  18. 2023Unifying Self-Supervised Clustering and Energy-Based Modelscited by 2
  19. 2026The Geometry of Projection Heads: Conditioning, Invariance, and Collapsecited by 1
  20. 2023On Linear Separation Capacity of Self-Supervised Representation Learningcited by 0
  21. 2024Direct Coloring for Self-Supervised Enhanced Feature Decouplingcited by 0
  22. 2025CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignmentcited by 0
  23. 2025Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimizationcited by 0
  24. 2025Learning the Neighborhood: Contrast-Free Multimodal Self-Supervised Molecular Graph Pretrainingcited by 0
  25. 2025Self-Supervised Representation Learning as Mutual Information Maximizationcited by 0
  26. 2026PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignmentcited by 0