mlsanityData to 13 Oct 2026

Projection Head is Secretly an Information Bottleneck

arXiv 2503.00507 ↗First posted 1 Mar 2025cs.LG cs.IT math.ITcited by 11

Not on the map. Why?

Abstract (arXiv)

Recently, contrastive learning has risen to be a promising paradigm for extracting meaningful data representations. Among various special designs, adding a projection head on top of the encoder during training and removing it for downstream tasks has proven to significantly enhance the performance of contrastive learning. However, despite its empirical success, the underlying mechanism of the projection head remains under-explored. In this paper, we develop an in-depth theoretical understanding of the projection head from the information-theoretic perspective. By establishing the theoretical guarantees on the downstream performance of the features before the projector, we reveal that an effective projector should act as an information bottleneck, filtering out the information irrelevant to the contrastive objective. Based on theoretical insights, we introduce modifications to projectors with training and structural regularizations. Empirically, our methods exhibit consistent improvement in the downstream performance across various real-world datasets, including CIFAR-10, CIFAR-100, and ImageNet-100. We believe our theoretical understanding on the role of the projection head will inspire more principled and advanced designs in this field. Code is available at https://github.com/PKU-ML/Projector_Theory.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 1 Mar 2025.

Builds on

35 of 39 references.

References · 35

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,730
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,860
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,826
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,397
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,033
  9. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 895
  10. 2020What Makes for Good Views for Contrastive Learning?cited by 844
  11. 2024Scaling and evaluating sparse autoencoderscited by 544
  12. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
23 more
  1. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 433
  2. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 375
  3. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 312
  4. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  5. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  6. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  7. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  8. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  9. 2022Understanding and Improving the Role of Projection Head in Self-Supervised Learningcited by 36
  10. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
  11. 2021Residual Relaxation for Multi-view Representation Learningcited by 30
  12. 2023On the Generalization of Multi-modal Contrastive Learningcited by 30
  13. 2022Simplicial Embeddings in Self-Supervised Learning and Downstream Classificationcited by 23
  14. 2022No Reason for No Supervision: Improved Generalization in Supervised Modelscited by 23
  15. 2023A Message Passing Perspective on Learning Dynamics of Contrastive Learningcited by 22
  16. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20
  17. 2023Rethinking Weak Supervision in Helping Contrastive Learningcited by 18
  18. 2024Non-negative Contrastive Learningcited by 18
  19. 2023Information Flow in Self-Supervised Learningcited by 16
  20. 2023Understanding Masked Autoencoders From a Local Contrastive Perspectivecited by 10
  21. 2024On the Role of Discrete Tokenization in Visual Representation Learningcited by 7
  22. 2023Unraveling Projection Heads in Contrastive Learning: Insights from Expansion and Shrinkagecited by 6
  23. 2023Deciphering the Projection Head: Representation Evaluation Self-supervised Learningcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learningcited by 5
  2. 2025An Augmentation Overlap Theory of Contrastive Learningcited by 5
  3. 2026LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneckcited by 5
  4. 2025AdaDim: Dimensionality Adaptation for SSL Representational Dynamicscited by 4
  5. 2026Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learningcited by 3
  6. 2025On the Alignment Between Supervised and Self-Supervised Contrastive Learningcited by 2
  7. 2026The Geometry of Projection Heads: Conditioning, Invariance, and Collapsecited by 1
  8. 2025CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignmentcited by 0
  9. 2026IB-HFN: Information Bottleneck-Driven SAR-Optical Fusion Network for High-Fidelity Cloud Removalcited by 0
  10. 2026A Theoretical Framework for Masked Pretraining (MPT)cited by 0
  11. 2026On the Role of the Projector in Contrastive Self-Supervised Learning: Last-Layer Rank Dynamics Drive Representation Qualitycited by 0