mlsanityData to 7 Oct 2026

Towards the Sparseness of Projection Head in Self-Supervised Learning

arXiv 2307.08913 ↗First posted 18 Jul 2023cs.LG cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

In recent years, self-supervised learning (SSL) has emerged as a promising approach for extracting valuable representations from unlabeled data. One successful SSL method is contrastive learning, which aims to bring positive examples closer while pushing negative examples apart. Many current contrastive learning approaches utilize a parameterized projection head. Through a combination of empirical analysis and theoretical investigation, we provide insights into the internal mechanisms of the projection head and its relationship with the phenomenon of dimensional collapse. Our findings demonstrate that the projection head enhances the quality of representations by performing contrastive loss in a projected subspace. Therefore, we propose an assumption that only a subset of features is necessary when minimizing the contrastive loss of a mini-batch of data. Theoretical analysis further suggests that a sparse projection head can enhance generalization, leading us to introduce SparseHead - a regularization term that effectively constrains the sparsity of the projection head, and can be seamlessly integrated with any self-supervised learning (SSL) approaches. Our experimental results validate the effectiveness of SparseHead, demonstrating its ability to improve the performance of existing contrastive methods.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 49 of its 63 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

49 of 63 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2017Mask R-CNNcited by 7,259
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
37 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  2. 2019Contrastive Multiview Codingcited by 1,297
  3. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  4. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  5. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  6. 2018Do Better ImageNet Models Transfer Better?cited by 560
  7. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  8. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  9. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  10. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  11. 2020Debiased Contrastive Learningcited by 354
  12. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  13. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  14. 2020Whitening for Self-Supervised Representation Learningcited by 278
  15. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  16. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  17. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  18. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  19. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  20. 2020On Linear Identifiability of Learned Representationscited by 103
  21. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  22. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  23. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  24. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  25. 2021Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptationcited by 50
  26. 2021Improving Transferability of Representations via Augmentation-Aware Self-Supervisioncited by 48
  27. 2021Robust Contrastive Learning Using Negative Samples with Diminished Semanticscited by 43
  28. 2022Do More Negative Samples Necessarily Hurt in Contrastive Learning?cited by 39
  29. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  30. 2022Self-Supervised Learning via Maximum Entropy Codingcited by 33
  31. 2022An Investigation into Whitening Loss for Self-supervised Learningcited by 21
  32. 2022MetAug: Contrastive Learning via Meta Feature Augmentationcited by 15
  33. 2023Unsupervised Meta-Learning via Few-shot Pseudo-supervised Contrastive Learningcited by 15
  34. 2022An Empirical Study on Disentanglement of Negative-free Contrastive Learningcited by 12
  35. 2021Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Frameworkcited by 11
  36. 2022Interventional Contrastive Learning with Meta Semantic Regularizercited by 11
  37. 2022MetaMask: Revisiting Dimensional Confounder for Self-Supervised Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024HypeBoy: Generative Self-Supervised Representation Learning on Hypergraphscited by 15
  2. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  3. 2026Radial Basis Function Networks as Projection Heads in Self-Supervised Learningcited by 0
  4. 2026On the Role of the Projector in Contrastive Self-Supervised Learning: Last-Layer Rank Dynamics Drive Representation Qualitycited by 0