mlsanityData to 7 Oct 2026

Deciphering the Projection Head: Representation Evaluation Self-supervised Learning

arXiv 2301.12189 ↗First posted 28 Jan 2023cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) aims to learn intrinsic features without labels. Despite the diverse architectures of SSL methods, the projection head always plays an important role in improving the performance of the downstream task. In this work, we systematically investigate the role of the projection head in SSL. Specifically, the projection head targets the uniformity part of SSL, which pushes the dissimilar samples away from each other, thus enabling the encoder to focus on extracting semantic features. Based on this understanding, we propose a Representation Evaluation Design (RED) in SSL models in which a shortcut connection between the representation and the projection vectors is built. Extensive experiments with different architectures, including SimCLR, MoCo-V2, and SimSiam, on various datasets, demonstrate that the representation evaluation design can consistently improve the baseline models in the downstream tasks. The learned representation from the RED-SSL models shows superior robustness to unseen augmentations and out-of-distribution data.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 48 references that are in the map. First posted on arXiv 28 Jan 2023.

Builds on

42 of 48 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2020Supervised Contrastive Learningcited by 2,803
30 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  8. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  9. 2020Understanding the Behaviour of Contrastive Losscited by 397
  10. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  11. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  12. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  13. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  14. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  15. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  16. 2021Decoupled Contrastive Learningcited by 141
  17. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  18. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  19. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  20. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  21. 2022Understanding Collapse in Non-Contrastive Siamese Representation Learningcited by 42
  22. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  23. 2021Revisiting the Transferability of Supervised Pretraining: an MLP Perspectivecited by 36
  24. 2022Understanding and Improving the Role of Projection Head in Self-Supervised Learningcited by 36
  25. 2021Exploring Set Similarity for Dense Self-supervised Representation Learningcited by 29
  26. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  27. 2020Towards Good Practices in Self-supervised Representation Learningcited by 19
  28. 2022Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learningcited by 19
  29. 2020Architectural Adversarial Robustness: The Case for Deep Pursuitcited by 18
  30. 2019Robust learning with implicit residual networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2024The Surprising Effectiveness of Skip-Tuning in Diffusion Samplingcited by 13
  2. 2025Projection Head is Secretly an Information Bottleneckcited by 11
  3. 2025Clustering Properties of Self-Supervised Learningcited by 8
  4. 2024Towards evolution of Deep Neural Networks through contrastive Self-Supervised learningcited by 0
  5. 2026Radial Basis Function Networks as Projection Heads in Self-Supervised Learningcited by 0