mlsanityData to 7 Oct 2026

On the Stepwise Nature of Self-Supervised Learning

arXiv 2303.15438 ↗First posted 27 Mar 2023cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

We present a simple picture of the training process of joint embedding self-supervised learning methods. We find that these methods learn their high-dimensional embeddings one dimension at a time in a sequence of discrete, well-separated steps. We arrive at this conclusion via the study of a linearized model of Barlow Twins applicable to the case in which the trained network is infinitely wide. We solve the training dynamics of this model from small initialization, finding that the model learns the top eigenmodes of a certain contrastive kernel in a stepwise fashion, and obtain a closed-form expression for the final learned representations. Remarkably, we then see the same stepwise learning phenomenon when training deep ResNets using the Barlow Twins, SimCLR, and VICReg losses. Our theory suggests that, just as kernel regression can be thought of as a model of supervised learning, kernel PCA may serve as a useful model of self-supervised learning.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 75 of its 90 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

75 of 90 references.

References · 75

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
38 more
  1. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  4. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  7. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  9. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  11. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  12. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  13. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  14. 2017Deep Neural Networks as Gaussian Processescited by 617
  15. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  16. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  17. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  18. 2017Large Batch Training of Convolutional Networkscited by 529
  19. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  20. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  21. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  22. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  23. 2017Implicit Regularization in Matrix Factorizationcited by 318
  24. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  25. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  26. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  27. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  28. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  29. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  30. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  31. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  32. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  33. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  34. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  35. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  36. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  37. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  38. 2020Intriguing Properties of Contrastive Lossescited by 129

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 15; 2025: 12; 2026: 5.

36 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  3. 2023Transformers learn through gradual rank increasecited by 52
  4. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
  5. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 33
  6. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 33
  7. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 30
  8. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 29
  9. 2026When Does LeJEPA Learn a World Model?cited by 27
  10. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  11. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  12. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
24 more
  1. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 11
  2. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  3. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  4. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  5. 2024Phase Transitions in the Output Distribution of Large Language Modelscited by 7
  6. 2025Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Modelscited by 7
  7. 2023Harnessing small projectors and multiple views for efficient vision pretrainingcited by 5
  8. 2024Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depthcited by 4
  9. 2024A Theoretical Characterization of Optimal Data Augmentations in Self-Supervised Learningcited by 4
  10. 2024Infinite Width Limits of Self Supervised Neural Networkscited by 4
  11. 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
  12. 2023Representation Learning Dynamics of Self-Supervised Modelscited by 3
  13. 2024Three Mechanisms of Feature Learning in a Linear Networkcited by 3
  14. 2025AdaDim: Dimensionality Adaptation for SSL Representational Dynamicscited by 3
  15. 2023FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learningcited by 2
  16. 2025Kernel VICReg for Self-Supervised Learning in Reproducing Kernel Hilbert Spacecited by 2
  17. 2024When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?cited by 1
  18. 2024Training Dynamics of Nonlinear Contrastive Learning Model in the High Dimensional Limitcited by 1
  19. 2024On the Generalization and Causal Explanation in Self-Supervised Learningcited by 0
  20. 2024Bi-ICE: An Inner Interpretable Framework for Image Classification via Bi-directional Interactions between Concept and Input Embeddingscited by 0
  21. 2025Interpreting Emergent Features in Deep Learning-based Side-channel Analysiscited by 0
  22. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  23. 2026An Analysis of Self-supervised Pre-training with Dependent Samplescited by 0
  24. 2026Neural Dynamics as the Composition of Quantized Unitscited by 0