# Dissecting Supervised Contrastive Learning

arXiv: https://arxiv.org/abs/2102.08817 · HTML: https://mlsanity.com/paper/2102.08817/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Characterizing Implicit Bias in Terms of Optimization Geometry](https://mlsanity.com/paper/1802.08246/) (2018)
- [NormFace: L2 Hypersphere Embedding for Face Verification](https://mlsanity.com/paper/1704.06369/) (2017)
- [Self-supervised Co-training for Video Representation Learning](https://mlsanity.com/paper/2010.09709/) (2020)
- [Convergence of Gradient Descent on Separable Data](https://mlsanity.com/paper/1803.01905/) (2018)
- [Analyzing and Improving Representations with the Soft Nearest Neighbor Loss](https://mlsanity.com/paper/1902.01889/) (2019)
- [Hyperspherical Prototype Networks](https://mlsanity.com/paper/1901.10514/) (2019)
- [Fix your classifier: the marginal value of training the last weight layer](https://mlsanity.com/paper/1801.04540/) (2018)

## Built on by (most cited first, 30 of 64)

- [SeeABLE: Soft Discrepancies and Bounded Contrastive Learning for Exposing Deepfakes](https://mlsanity.com/paper/2211.11296/) (2022)
- [Understanding Imbalanced Semantic Segmentation Through Neural Collapse](https://mlsanity.com/paper/2301.01100/) (2023)
- [Improving Self-Supervised Learning by Characterizing Idealized Representations](https://mlsanity.com/paper/2209.06235/) (2022)
- [A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm](https://mlsanity.com/paper/2310.12244/) (2023)
- [Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes](https://mlsanity.com/paper/2410.23126/) (2024)
- [CoFInAl: Enhancing Action Quality Assessment with Coarse-to-Fine Instruction Alignment](https://mlsanity.com/paper/2404.13999/) (2024)
- [Contrastive Unlearning: A Contrastive Approach to Machine Unlearning](https://mlsanity.com/paper/2401.10458/) (2024)
- [Contrastive learning for regression in multi-site brain age prediction](https://mlsanity.com/paper/2211.08326/) (2022)
- [Neural Collapse in Multi-label Learning with Pick-all-label Loss](https://mlsanity.com/paper/2310.15903/) (2023)
- [Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame](https://mlsanity.com/paper/2411.01248/) (2024)
- [On Measuring Excess Capacity in Neural Networks](https://mlsanity.com/paper/2202.08070/) (2022)
- [ACTION++: Improving Semi-supervised Medical Image Segmentation with Adaptive Anatomical Contrast](https://mlsanity.com/paper/2304.02689/) (2023)
- [Mini-Batch Optimization of Contrastive Loss](https://mlsanity.com/paper/2307.05906/) (2023)
- [Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss](https://mlsanity.com/paper/2310.00893/) (2023)
- [DELTA: Decoupling Long-Tailed Online Continual Learning](https://mlsanity.com/paper/2404.04476/) (2024)
- [Generalizable Person Re-identification via Balancing Alignment and Uniformity](https://mlsanity.com/paper/2411.11471/) (2024)
- [Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking](https://mlsanity.com/paper/2509.17738/) (2025)
- [A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalization](https://mlsanity.com/paper/2304.08914/) (2023)
- [Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges](https://mlsanity.com/paper/2310.08358/) (2023)
- [Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses](https://mlsanity.com/paper/2405.18045/) (2024)
- [Efficient and Versatile Robust Fine-Tuning of Zero-shot Models](https://mlsanity.com/paper/2408.05749/) (2024)
- [Cell-ontology guided transcriptome foundation model](https://mlsanity.com/paper/2408.12373/) (2024)
- [Faithful and Consistent Graph Neural Network Explanations with Rationale Alignment](https://mlsanity.com/paper/2301.02791/) (2023)
- [Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse](https://mlsanity.com/paper/2311.05139/) (2023)
- [Learn From Zoom: Decoupled Supervised Contrastive Learning For WCE Image Classification](https://mlsanity.com/paper/2401.05771/) (2024)
- [Analysis of Using Sigmoid Loss for Contrastive Learning](https://mlsanity.com/paper/2402.12613/) (2024)
- [Consistent Supervised-Unsupervised Alignment for Generalized Category Discovery](https://mlsanity.com/paper/2507.04725/) (2025)
- [SC-MIL: Supervised Contrastive Multiple Instance Learning for Imbalanced Classification in Pathology](https://mlsanity.com/paper/2303.13405/) (2023)
- [Towards understanding neural collapse in supervised contrastive learning with the information bottleneck method](https://mlsanity.com/paper/2305.11957/) (2023)
- [A Tale of Two Classes: Adapting Supervised Contrastive Learning to Binary Imbalanced Datasets](https://mlsanity.com/paper/2503.17024/) (2025)

## Abstract (arXiv)

Minimizing cross-entropy over the softmax scores of a linear map composed with a high-capacity encoder is arguably the most popular choice for training neural networks on supervised learning tasks. However, recent works show that one can directly optimize the encoder instead, to obtain equally (or even more) discriminative representations via a supervised variant of a contrastive objective. In this work, we address the question whether there are fundamental differences in the sought-for representation geometry in the output space of the encoder at minimal loss. Specifically, we prove, under mild assumptions, that both losses attain their minimum once the representations of each class collapse to the vertices of a regular simplex, inscribed in a hypersphere. We provide empirical evidence that this configuration is attained in practice and that reaching a close-to-optimal state typically indicates good generalization performance. Yet, the two losses show remarkably different optimization behavior. The number of iterations required to perfectly fit to data scales superlinearly with the amount of randomly flipped labels for the supervised contrastive loss. This is in contrast to the approximately linear scaling previously reported for networks trained with cross-entropy.