# A Probabilistic Model for Non-Contrastive Learning

arXiv: https://arxiv.org/abs/2501.13031 · HTML: https://mlsanity.com/paper/2501.13031/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 13 of 13)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning](https://mlsanity.com/paper/2105.15134/) (2021)
- [Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning](https://mlsanity.com/paper/2106.09226/) (2021)
- [Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism](https://mlsanity.com/paper/2303.02387/) (2023)
- [Contrasting the landscape of contrastive and non-contrastive learning](https://mlsanity.com/paper/2203.15702/) (2022)
- [Probabilistic Self-supervised Learning via Scoring Rules Minimization](https://mlsanity.com/paper/2309.02048/) (2023)
- [Bayesian Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2301.11673/) (2023)
- [Representation Learning Dynamics of Self-Supervised Models](https://mlsanity.com/paper/2309.02011/) (2023)

## Abstract (arXiv)

Self-supervised learning (SSL) aims to find meaningful representations from unlabeled data by encoding semantic similarities through data augmentations. Despite its current popularity, theoretical insights about SSL are still scarce. For example, it is not yet known whether commonly used SSL loss functions can be related to a statistical model, much in the same as OLS, generalized linear models or PCA naturally emerge as maximum likelihood estimates of an underlying generative process. In this short paper, we consider a latent variable statistical model for SSL that exhibits an interesting property: Depending on the informativeness of the data augmentations, the MLE of the model either reduces to PCA, or approaches a simple non-contrastive loss. We analyze the model and also empirically illustrate our findings.