mlsanityData to 7 Oct 2026

What Do We Maximize in Self-Supervised Learning?

arXiv 2207.10081 ↗First posted 20 Jul 2022cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

In this paper, we examine self-supervised learning methods, particularly VICReg, to provide an information-theoretical understanding of their construction. As a first step, we demonstrate how information-theoretic quantities can be obtained for a deterministic network, offering a possible alternative to prior work that relies on stochastic models. This enables us to demonstrate how VICReg can be (re)discovered from first principles and its assumptions about data distribution. Furthermore, we empirically demonstrate the validity of our assumptions, confirming our novel understanding of VICReg. Finally, we believe that the derivation and insights we obtain can be generalized to many other SSL methods, opening new avenues for theoretical and practical understanding of SSL and transfer learning.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 20 Jul 2022.

Builds on

27 of 42 references.

References · 27

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  9. 2016Deep Variational Information Bottleneckcited by 915
  10. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  11. 2019On Variational Bounds of Mutual Informationcited by 550
  12. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
15 more
  1. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  2. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  3. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  4. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  5. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  6. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 145
  7. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  8. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  9. 2021Lossy Compression for Lossless Predictioncited by 51
  10. 2021Compressive Visual Representationscited by 41
  11. 2017Estimating Mixture Entropy with Pairwise Distancescited by 34
  12. 2022Information Flow in Deep Neural Networkscited by 22
  13. 2019Information in Infinite Ensembles of Infinitely-Wide Neural Networkscited by 14
  14. 2020The Dual Information Bottleneckcited by 14
  15. 2018Estimating Information Flow in Deep Neural Networkscited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 8; 2024: 2; 2025: 4.

15 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  3. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  4. 2023Reverse Engineering Self-Supervised Learningcited by 33
  5. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  6. 2024DINO as a von Mises-Fisher mixture modelcited by 14
  7. 2023Variance-Covariance Regularization Improves Representation Learningcited by 13
  8. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  9. 2022Minimalistic Unsupervised Learning with the Sparse Manifold Transformcited by 6
  10. 2025Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learningcited by 4
  11. 2023Semi-Supervised Semantic Segmentation of Cell Nuclei via Diffusion-based Large-Scale Pre-Training and Collaborative Learningcited by 2
  12. 2023Feature propagation as self-supervision signals on graphscited by 0
3 more
  1. 2023Layer-wise training for self-supervised learning on graphscited by 0
  2. 2024Leveraging Self-Supervised Learning for Fetal Cardiac Planes Classification using Ultrasound Scan Videoscited by 0
  3. 2025Scaling Non-Parametric Sampling with Representationcited by 0