mlsanityData to 7 Oct 2026

The SSL Interplay: Augmentations, Inductive Bias, and Generalization

arXiv 2302.02774 ↗First posted 6 Feb 2023stat.ML cs.AI cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has emerged as a powerful framework to learn representations from raw data without supervision. Yet in practice, engineers face issues such as instability in tuning optimizers and collapse of representations during training. Such challenges motivate the need for a theory to shed light on the complex interplay between the choice of data augmentation, network architecture, and training algorithm. We study such an interplay with a precise analysis of generalization performance on both pretraining and downstream tasks in a theory friendly setup, and highlight several insights for SSL practitioners that arise from our theory.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 42 of its 80 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

42 of 80 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2014Convex Optimization: Algorithms and Complexitycited by 569
  11. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  12. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
30 more
  1. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  2. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  3. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  4. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  5. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  6. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  7. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  8. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  9. 2016A vector-contraction inequality for Rademacher complexitiescited by 141
  10. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  11. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  12. 2021Learning with invariances in random features and kernel modelscited by 89
  13. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  14. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  15. 2018Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spacescited by 59
  16. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  17. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 38
  18. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  19. 2018Finite-sample analysis of M-estimators using self-concordancecited by 30
  20. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  21. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  22. 2021Distribution-Free Robust Linear Regressioncited by 19
  23. 2022Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learningcited by 19
  24. 2022An elementary analysis of ridge regression with random designcited by 15
  25. 2020Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learningcited by 14
  26. 2014The geometry of kernelized spectral clusteringcited by 13
  27. 2021On the Sample Complexity of Learning under Invariance and Geometric Stabilitycited by 13
  28. 2022Joint Embedding Self-Supervised Learning in the Kernel Regimecited by 12
  29. 2021Fast rates in structured predictioncited by 11
  30. 2022On minimal variations for unsupervised representation learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 11; 2025: 10; 2026: 4.

40 papers.

Sort:

  1. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  2. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 33
  3. 2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29
  4. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  5. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  6. 2024A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classificationcited by 19
  7. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  8. 2023Information Flow in Self-Supervised Learningcited by 16
  9. 2023Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Needcited by 9
  10. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  11. 2023The Galerkin method beats Graph-Based Approaches for Spectral Algorithmscited by 6
  12. 2024Tight PAC-Bayesian Risk Certificates for Contrastive Learningcited by 6
28 more
  1. 2023Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shiftcited by 5
  2. 2023Harnessing small projectors and multiple views for efficient vision pretrainingcited by 5
  3. 2025Why and How Auxiliary Tasks Improve JEPA Representationscited by 5
  4. 2024A Theoretical Characterization of Optimal Data Augmentations in Self-Supervised Learningcited by 4
  5. 2024Infinite Width Limits of Self Supervised Neural Networkscited by 4
  6. 2023Novel Categories Discovery Via Constraints on Empirical Prediction Statisticscited by 3
  7. 2023Representation Learning Dynamics of Self-Supervised Modelscited by 3
  8. 2023Training Dynamics of Deep Network Linear Regionscited by 3
  9. 2024GPS-SSL: Guided Positive Sampling to Inject Prior Into Self-Supervised Learningcited by 3
  10. 2024Disentangling Masked Autoencoders for Unsupervised Domain Generalizationcited by 3
  11. 2023Self-Supervision for Tackling Unsupervised Anomaly Detection: Pitfalls and Opportunitiescited by 2
  12. 2024Spectrally Transformed Kernel Regressioncited by 2
  13. 2024PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesiscited by 2
  14. 2024OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?cited by 2
  15. 2026Global Geometry Is Not Enough for Vision Representationscited by 2
  16. 2024When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?cited by 1
  17. 2025Understanding Learning Dynamics Through Structured Representationscited by 1
  18. 2025Interpretable Self-Supervised Learning via Representer Landmarks and Nystr"om Approximationcited by 1
  19. 2026When to Align, When to Predict: A Phase Diagram for Multimodal Learningcited by 1
  20. 2023On Linear Separation Capacity of Self-Supervised Representation Learningcited by 0
  21. 2023Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representationscited by 0
  22. 2025ConceptVAE: Self-Supervised Fine-Grained Concept Disentanglement from 2D Echocardiographiescited by 0
  23. 2025Exploring Disparity-Accuracy Trade-offs in Face Recognition Systems: The Role of Datasets, Architectures, and Loss Functionscited by 0
  24. 2025The Efficacy of Semantics-Preserving Transformations in Self-Supervised Learning for Medical Ultrasoundcited by 0
  25. 2025Enhancing VICReg: Random-Walk Pairing for Improved Generalization and Better Global Semantics Capturingcited by 0
  26. 2025Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimizationcited by 0
  27. 2026A Schr"odinger Eigenfunction Method for Long-Horizon Stochastic Optimal Controlcited by 0
  28. 2026An Analysis of Self-supervised Pre-training with Dependent Samplescited by 0