mlsanityData to 14 Oct 2026

A Probabilistic Model for Non-Contrastive Learning

arXiv 2501.13031 ↗First posted 22 Jan 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) aims to find meaningful representations from unlabeled data by encoding semantic similarities through data augmentations. Despite its current popularity, theoretical insights about SSL are still scarce. For example, it is not yet known whether commonly used SSL loss functions can be related to a statistical model, much in the same as OLS, generalized linear models or PCA naturally emerge as maximum likelihood estimates of an underlying generative process. In this short paper, we consider a latent variable statistical model for SSL that exhibits an interesting property: Depending on the informativeness of the data augmentations, the MLE of the model either reduces to PCA, or approaches a simple non-contrastive loss. We analyze the model and also empirically illustrate our findings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 22 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

13 of 22 references.

References · 13

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  4. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  5. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  6. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  7. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  8. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  9. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 33
  10. 2022Contrasting the landscape of contrastive and non-contrastive learningcited by 24
  11. 2023Probabilistic Self-supervised Learning via Scoring Rules Minimizationcited by 4
  12. 2023Bayesian Self-Supervised Contrastive Learningcited by 3
1 more
  1. 2023Representation Learning Dynamics of Self-Supervised Modelscited by 3

Built on by

Not cited yet.