mlsanityData to 7 Oct 2026

Understanding and Improving the Role of Projection Head in Self-Supervised Learning

arXiv 2212.11491 ↗First posted 22 Dec 2022cs.LG cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) aims to produce useful feature representations without access to any human-labeled data annotations. Due to the success of recent SSL methods based on contrastive learning, such as SimCLR, this problem has gained popularity. Most current contrastive learning approaches append a parametrized projection head to the end of some backbone network to optimize the InfoNCE objective and then discard the learned projection head after training. This raises a fundamental question: Why is a learnable projection head required if we are to discard it after training? In this work, we first perform a systematic study on the behavior of SSL training focusing on the role of the projection head layers. By formulating the projection head as a parametric component for the InfoNCE objective rather than a part of the network, we present an alternative optimization scheme for training contrastive learning based SSL frameworks. Our experimental study on multiple image classification datasets demonstrates the effectiveness of the proposed approach over alternatives in the SSL literature.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 22 Dec 2022.

Builds on

26 of 30 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
14 more
  1. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  2. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  3. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  4. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  5. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  6. 2020Debiased Contrastive Learningcited by 354
  7. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  8. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  9. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  10. 2020Whitening for Self-Supervised Representation Learningcited by 278
  11. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  12. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  13. 2020Intriguing Properties of Contrastive Lossescited by 129
  14. 2020BYOL works even without batch statisticscited by 95

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 8; 2025: 11; 2026: 3.

36 papers.

Sort:

By area:

  1. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  2. 2023On the Embedding Collapse when Scaling up Recommendation Modelscited by 29
  3. 2024Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learningcited by 25
  4. 2025RANGE: Retrieval Augmented Neural Fields for Multi-Resolution Geo-Embeddingscited by 18
  5. 2023Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognitioncited by 14
  6. 2025Projection Head is Secretly an Information Bottleneckcited by 11
  7. 2023Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusioncited by 10
  8. 2025Clustering Properties of Self-Supervised Learningcited by 8
  9. 2024ViLReF: An Expert Knowledge Enabled Vision-Language Retinal Foundation Modelcited by 7
  10. 2025PEER pressure: Model-to-Model Regularization for Single Source Domain Generalizationcited by 7
  11. 2023Evaluating Self-Supervised Learning via Risk Decompositioncited by 6
  12. 2023Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learningcited by 6
24 more
  1. 2023Deciphering the Projection Head: Representation Evaluation Self-supervised Learningcited by 5
  2. 2023Data Compression and Inference in Cosmology with Self-Supervised Machine Learningcited by 5
  3. 2024A Probabilistic Model Behind Self-Supervised Learningcited by 5
  4. 2024Beyond Pairwise Correlations: Higher-Order Redundancies in Self-Supervised Representation Learningcited by 4
  5. 2025A Simple Contrastive Framework Of Item Tokenization For Generative Recommendationcited by 4
  6. 2024Joint-Embedding Masked Autoencoder for Self-supervised Learning of Dynamic Functional Connectivity from the Human Braincited by 3
  7. 2024BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimizationcited by 3
  8. 2023S-JEA: Stacked Joint Embedding Architectures for Self-Supervised Visual Representation Learningcited by 2
  9. 2023Look-Ahead Selective Plasticity for Continual Learning of Visual Taskscited by 2
  10. 2025On the Alignment Between Supervised and Self-Supervised Contrastive Learningcited by 2
  11. 2024Enhancing Cardiovascular Disease Prediction through Multi-Modal Self-Supervised Learningcited by 1
  12. 2025Cross-Modal Consistency Learning for Sign Language Recognitioncited by 1
  13. 2023Generate to Understand for Representationcited by 0
  14. 2023SemiSFL: Split Federated Learning on Unlabeled and Non-IID Datacited by 0
  15. 2023Stabilizing Subject Transfer in EEG Classification with Divergence Estimationcited by 0
  16. 2023Unveiling the Potential of Probabilistic Embeddings in Self-Supervised Learningcited by 0
  17. 2024An Empirical Study on Improving SimCLR's Nonlinear Projection Head using Pretrained Autoencoder Embeddingscited by 0
  18. 2025PacketCLIP: Multi-Modal Embedding of Network Traffic and Language for Cybersecurity Reasoningcited by 0
  19. 2025Representation learning with a transformer by contrastive learning for money laundering detectioncited by 0
  20. 2025CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignmentcited by 0
  21. 2025ANCHOR: Integrating Adversarial Training with Hard-mined Supervised Contrastive Learning for Robust Representation Learningcited by 0
  22. 2026Radial Basis Function Networks as Projection Heads in Self-Supervised Learningcited by 0
  23. 2026Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Executioncited by 0
  24. 2026Which Tasks Survive Self-Supervised Learning?cited by 0