mlsanityData to 7 Oct 2026

Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features

arXiv 2402.18884 ↗First posted 29 Feb 2024cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent findings reveal that over-parameterized deep neural networks, trained beyond zero training-error, exhibit a distinctive structural pattern at the final layer, termed as Neural-collapse (NC). These results indicate that the final hidden-layer outputs in such networks display minimal within-class variations over the training set. While existing research extensively investigates this phenomenon under cross-entropy loss, there are fewer studies focusing on its contrastive counterpart, supervised contrastive (SC) loss. Through the lens of NC, this paper employs an analytical approach to study the solutions derived from optimizing the SC loss. We adopt the unconstrained features model (UFM) as a representative proxy for unveiling NC-related phenomena in sufficiently over-parameterized deep networks. We show that, despite the non-convexity of SC loss minimization, all local minima are global minima. Furthermore, the minimizer is unique (up to a rotation). We prove our results by formalizing a tight convex relaxation of the UFM. Finally, through this convex formulation, we delve deeper into characterizing the properties of global solutions under label-imbalanced training data.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 27 of its 33 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

27 of 33 references.

References · 27

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2020Supervised Contrastive Learningcited by 2,803
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  4. 2019Contrastive Multiview Codingcited by 1,297
  5. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  8. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  9. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  10. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  11. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  12. 2020Neural collapse with unconstrained featurescited by 140
15 more
  1. 2021Targeted Supervised Contrastive Learning for Long-Tailed Recognitioncited by 132
  2. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  3. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  4. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  5. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  6. 2021Distributional Robustness Loss for Long-tail Learningcited by 84
  7. 2018Risk and parameter convergence of logistic regressioncited by 81
  8. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  9. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  10. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  11. 2020Neural Collapse with Cross-Entropy Losscited by 52
  12. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  13. 2022Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learningcited by 43
  14. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  15. 2022ELM: Embedding and Logit Margins for Long-Tail Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 4.

5 papers.

  1. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  2. 2026Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasetscited by 0
  3. 2026A Unified Geometric Framework for Weighted Contrastive Learningcited by 0
  4. 2026Multi-Marginal Inverse Optimal Transport for Contrastive Learning Via Explicit Anchor-Positive-Negative Couplingcited by 0
  5. 2026Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysiscited by 0