mlsanityData to 7 Oct 2026

Improving Self-Supervised Learning by Characterizing Idealized Representations

arXiv 2209.06235 ↗First posted 13 Sep 2022cs.LG stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

Despite the empirical successes of self-supervised learning (SSL) methods, it is unclear what characteristics of their representations lead to high downstream accuracies. In this work, we characterize properties that SSL representations should ideally satisfy. Specifically, we prove necessary and sufficient conditions such that for any task invariant to given data augmentations, desired probes (e.g., linear or MLP) trained on that representation attain perfect accuracy. These requirements lead to a unifying conceptual framework for improving existing SSL methods and deriving new ones. For contrastive learning, our framework prescribes simple but significant improvements to previous methods such as using asymmetric projection heads. For non-contrastive learning, we use our framework to derive a simple and novel objective. Our resulting SSL algorithms outperform baselines on standard benchmarks, including SwAV+multicrops on linear probing of ImageNet.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 84 of its 139 references that are in the map. First posted on arXiv 13 Sep 2022.

Builds on

84 of 139 references.

References · 84

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
38 more
  1. 2021Convex optimizationcited by 2,479
  2. 2013Describing Textures in the Wildcited by 2,110
  3. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2019Contrastive Multiview Codingcited by 1,297
  8. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  9. 2016Colorful Image Colorizationcited by 1,113
  10. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  13. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  14. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  15. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  16. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  17. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  18. 2020Understanding the Behaviour of Contrastive Losscited by 397
  19. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  20. 2019On Mutual Information Maximization for Representation Learningcited by 315
  21. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  22. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  23. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  24. 2020Whitening for Self-Supervised Representation Learningcited by 278
  25. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  26. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  27. 2016Joint Unsupervised Learning of Deep Representations and Image Clusterscited by 205
  28. 2020Representation Learning via Invariant Causal Mechanismscited by 197
  29. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  30. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  31. 2016Exploring the Limits of Language Modelingcited by 172
  32. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  33. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  34. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  35. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  36. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  37. 2020Neural collapse with unconstrained featurescited by 140
  38. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 18; 2024: 6; 2025: 4; 2026: 1.

33 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2023No Representation Rules Them All in Category Discoverycited by 61
  3. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
  4. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 38
  5. 2023Reverse Engineering Self-Supervised Learningcited by 33
  6. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 33
  7. 2023Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representationscited by 31
  8. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  9. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  10. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  11. 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 16
  12. 2023Information Flow in Self-Supervised Learningcited by 16
21 more
  1. 2022Implicit variance regularization in non-contrastive SSLcited by 15
  2. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  3. 2024LDReg: Local Dimensionality Regularized Self-Supervised Learningcited by 11
  4. 2023Quantifying the Variability Collapse of Neural Networkscited by 9
  5. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 9
  6. 2023Intrinsic Self-Supervision for Data Quality Auditscited by 8
  7. 2024Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?cited by 8
  8. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  9. 2022GEDI: GEnerative and DIscriminative Training for Self-Supervised Learningcited by 6
  10. 2023Evaluating Self-Supervised Learning via Risk Decompositioncited by 6
  11. 2023Substance or Style: What Does Your Image Embedding Know?cited by 5
  12. 2023A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representationcited by 4
  13. 2023Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Headcited by 3
  14. 2024BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimizationcited by 3
  15. 2022SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Datacited by 2
  16. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
  17. 2023Unifying Self-Supervised Clustering and Energy-Based Modelscited by 2
  18. 2026A Fiber Criterion for Representation Identifiability in Supervised Learningcited by 2
  19. 2023How does Contrastive Learning Organize Images?cited by 1
  20. 2025Information-Maximized Soft Variable Discretization for Self-Supervised Image Representation Learningcited by 1
  21. 2023Deep Tensor Networkcited by 0