mlsanityData to 14 Oct 2026

Information theoretic underpinning of self-supervised learning by clustering

arXiv 2605.11870 ↗First posted 12 May 2026cs.LG cs.IT math.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) is recognized as an essential tool for building foundation models for Artificial Intelligence applications. The advances in SSL have been made thanks to vigorous arguments about the principles of SSL and through extensive empirical research. The aim of this paper is to contribute to the development of the underpinning theory of SSL, focusing on the deep clustering approach. By analogy to supervised learning, we formulate SSL as K-L divergence optimization. The mode collapse is prevented by imposing an optimisation constraint on the teacher distribution. This leads to normalization using inverse cluster priors. We show that using Jensen inequality this normalization simplifies to the popular batch centering procedure. Distillation and centering are common {heuristics-based} practices in SSL, {but our work underpins them theoretically.} The theoretical model developed not only supports specific existing successful SSL methods, but also suggests directions for future investigations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

37 of 49 references.

References · 37

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  12. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
25 more
  1. 2025DINOv3cited by 1,414
  2. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  3. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  4. 2016Colorful Image Colorizationcited by 1,114
  5. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  6. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  7. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  8. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  9. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  10. 2016Learning Representations for Automatic Colorizationcited by 251
  11. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  12. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  13. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  14. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  15. 2021SiT: Self-supervised vIsion Transformercited by 99
  16. 2017Unsupervised Learning by Predicting Noisecited by 98
  17. 2021Unsupervised Object-Level Representation Learning from Scene Imagescited by 72
  18. 2018Learning Image Representations by Completing Damaged Jigsaw Puzzlescited by 66
  19. 2018Self-Supervised Feature Learning by Learning to Spot Artifactscited by 43
  20. 2021Self-Supervised Classification Networkcited by 28
  21. 2023CrOC: Cross-View Online Clustering for Dense Visual Representation Learningcited by 21
  22. 2023CrIBo: Self-Supervised Learning via Cross-Image Object-Level Bootstrappingcited by 17
  23. 2023Information Flow in Self-Supervised Learningcited by 16
  24. 2025Object-Centric Pretraining via Target Encoder Bootstrappingcited by 6
  25. 2023Unifying Self-Supervised Clustering and Energy-Based Modelscited by 2

Built on by

Not cited yet.