mlsanityData to 14 Oct 2026

Understanding Self-supervised Contrastive Learning through Supervised Objectives

arXiv 2510.10572 ↗First posted 12 Oct 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-supervised representation learning has achieved impressive empirical success, yet its theoretical understanding remains limited. In this work, we provide a theoretical perspective by formulating self-supervised representation learning as an approximation to supervised representation learning objectives. Based on this formulation, we derive a loss function closely related to popular contrastive losses such as InfoNCE, offering insight into their underlying principles. Our derivation naturally introduces the concepts of prototype representation bias and a balanced contrastive loss, which help explain and improve the behavior of self-supervised learning algorithms. We further show how components of our theoretical framework correspond to established practices in contrastive learning. Finally, we empirically validate the effect of balancing positive and negative pair interactions. All theoretical proofs are provided in the appendix, and our code is included in the supplementary material.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 57 of its 72 references that are in the map. First posted on arXiv 12 Oct 2025.

Builds on

57 of 72 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2015Fast R-CNNcited by 2,933
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
38 more
  1. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  2. 2020Supervised Contrastive Learningcited by 2,829
  3. 2021Convex optimizationcited by 2,515
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  8. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  10. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  11. 2016Colorful Image Colorizationcited by 1,114
  12. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  13. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  14. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  15. 2017Large Batch Training of Convolutional Networkscited by 535
  16. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  17. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  18. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  19. 2020Understanding the Behaviour of Contrastive Losscited by 398
  20. 2020Debiased Contrastive Learningcited by 356
  21. 2020Are we done with ImageNet?cited by 333
  22. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  23. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  24. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  25. 2020What Should Not Be Contrastive in Contrastive Learningcited by 226
  26. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  27. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  28. 2021Decoupled Contrastive Learningcited by 143
  29. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  30. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  31. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  32. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  33. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  34. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 61
  35. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 58
  36. 2022Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCocited by 52
  37. 2023Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Datacited by 44
  38. 2022On the Importance of Asymmetry for Siamese Representation Learningcited by 39

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.