mlsanityData to 7 Oct 2026

Unraveling Projection Heads in Contrastive Learning: Insights from Expansion and Shrinkage

arXiv 2306.03335 ↗First posted 6 Jun 2023stat.ML cs.LG math.STcited by 6

Not on the map. Why?

Abstract (arXiv)

We investigate the role of projection heads, also known as projectors, within the encoder-projector framework (e.g., SimCLR) used in contrastive learning. We aim to demystify the observed phenomenon where representations learned before projectors outperform those learned after -- measured using the downstream linear classification accuracy, even when the projectors themselves are linear. In this paper, we make two significant contributions towards this aim. Firstly, through empirical and theoretical analysis, we identify two crucial effects -- expansion and shrinkage -- induced by the contrastive loss on the projectors. In essence, contrastive loss either expands or shrinks the signal direction in the representations learned by an encoder, depending on factors such as the augmentation strength, the temperature used in contrastive loss, etc. Secondly, drawing inspiration from the expansion and shrinkage phenomenon, we propose a family of linear transformations to accurately model the projector's behavior. This enables us to precisely characterize the downstream linear classification accuracy in the high-dimensional asymptotic limit. Our findings reveal that linear projectors operating in the shrinkage (or expansion) regime hinder (or improve) the downstream classification accuracy. This provides the first theoretical explanation as to why (linear) projectors impact the downstream performance of learned representations. Our theoretical findings are further corroborated by extensive experiments on both synthetic data and real image data.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 47 of its 60 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

47 of 60 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  11. 2014How transferable are features in deep neural networks?cited by 1,216
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
35 more
  1. 2019When Does Label Smoothing Help?cited by 985
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2019Benign Overfitting in Linear Regressioncited by 544
  7. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  8. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  9. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  10. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  11. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  12. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  13. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  14. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  15. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  16. 2023A Cookbook of Self-Supervised Learningcited by 248
  17. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  18. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  19. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  20. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  21. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  22. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  23. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  24. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  25. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  26. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  27. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  28. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  29. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  30. 2020A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifierscited by 44
  31. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  32. 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
  33. 2022Toward a Geometrical Understanding of Self-supervised Contrastive Learningcited by 17
  34. 2022Unsupervised visualization of image datasets using contrastive learningcited by 16
  35. 2021Tractability from overparametrization: The example of the negative perceptroncited by 12

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 3.

6 papers.

  1. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20
  2. 2025Projection Head is Secretly an Information Bottleneckcited by 11
  3. 2025Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variablescited by 7
  4. 2024DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representationscited by 2
  5. 2024An Empirical Study on Improving SimCLR's Nonlinear Projection Head using Pretrained Autoencoder Embeddingscited by 0
  6. 2025ANCHOR: Integrating Adversarial Training with Hard-mined Supervised Contrastive Learning for Robust Representation Learningcited by 0