mlsanityData to 14 Oct 2026

A Theory of Contrastive Learning with Natural Images

arXiv 2607.07470 ↗First posted 8 Jul 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Why does contrastive learning with simple images and augmentations yield useful representations for downstream tasks? We address this question by analytically computing the optimal representation in terms of a contrastive loss for a range of basic augmentations and any image dataset with stationary statistics. We show that for certain augmentations the optimum can be attained by a CNN whose first layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer that performs partial whitening. We also show that the optimal weights in such CNNs for more complicated augmentations are still sinusoids. The frequencies of the sinusoids and their weights can be computed using a simple waterfilling algorithm given the dataset's expected power spectrum. Experiments with different image datasets and augmentations show that such CNNs trained with SGD empirically learn sinusoids in their first layer and to perform partial whitening

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

18 of 33 references.

References · 18

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  5. 2020Supervised Contrastive Learningcited by 2,829
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  8. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  10. 2020Whitening for Self-Supervised Representation Learningcited by 279
  11. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  12. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
6 more
  1. 2021Learning to See by Looking at Noisecited by 92
  2. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  3. 2026InfoNCE Induces Gaussian Distributioncited by 16
  4. 2023Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activationscited by 11
  5. 2024Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularizationcited by 9
  6. 2024Understanding Self-Supervised Learning via Gaussian Mixture Modelscited by 6

Built on by

Not cited yet.