mlsanityData to 13 Oct 2026

Evaluating Self-Supervised Learning via Risk Decomposition

arXiv 2302.03068 ↗First posted 6 Feb 2023cs.LG cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) pipelines differ in many design choices such as the architecture, augmentations, or pretraining data. Yet SSL is typically evaluated using a single metric: linear probing on ImageNet. This does not provide much insight into why or when a model is better, now how to improve it. To address this, we propose an SSL risk decomposition, which generalizes the classical supervised approximation-estimation decomposition by considering errors arising from the representation learning step. Our decomposition consists of four error components: approximation, representation usability, probe generalization, and encoder generalization. We provide efficient estimators for each component and use them to analyze the effect of 30 design choices on 169 SSL vision models evaluated on ImageNet. Our analysis gives valuable insights for designing and using SSL models. For example, it highlights the main sources of error and shows how to improve SSL in specific settings (full- vs few-shot) by trading off error components. All results and pretrained models are at https://github.com/YannDubs/SSL-Risk-Decomposition.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 74 of its 102 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

74 of 102 references.

References · 74

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  11. 2017A Unified Approach to Interpreting Model Predictionscited by 6,515
  12. 2020Scaling Laws for Neural Language Modelscited by 6,494
38 more
  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,143
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,730
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,850
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,368
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,285
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,841
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,826
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,612
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,397
  10. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,381
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,378
  12. 2019Contrastive Multiview Codingcited by 1,297
  13. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,164
  14. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,064
  15. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,023
  16. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 920
  17. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 895
  18. 2020What Makes for Good Views for Contrastive Learning?cited by 844
  19. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  20. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  21. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 670
  22. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  23. 2019Self-labelling via simultaneous clustering and representation learningcited by 526
  24. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  25. 2020Understanding the Behaviour of Contrastive Losscited by 397
  26. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 375
  27. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 340
  28. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 312
  29. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  30. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  31. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 205
  32. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  33. 2020Representation Learning via Invariant Causal Mechanismscited by 197
  34. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 190
  35. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  36. 2020A Theory of Usable Information Under Computational Constraintscited by 156
  37. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  38. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 0; 2026: 1.

6 papers.

  1. 2024AutoBencher: Towards Declarative Benchmark Constructioncited by 42
  2. 2024Scaling Laws for the Value of Individual Data Points in Machine Learningcited by 10
  3. 2023Analyzing the Sample Complexity of Self-Supervised Image Reconstruction Methodscited by 8
  4. 2024CREMA: A Contrastive Regularized Masked Autoencoder for Robust ECG Diagnostics across Clinical Domainscited by 8
  5. 2024Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretrainingcited by 7
  6. 2026AudioMosaic: Contrastive Masked Audio Representation Learningcited by 1