mlsanityData to 7 Oct 2026

What shapes the loss landscape of self-supervised learning?

arXiv 2210.00638 ↗First posted 2 Oct 2022cs.LG physics.data-ancited by 19

Not on the map. Why?

Abstract (arXiv)

Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL). However, questions remain in our theoretical understanding: When do those collapses occur? What are the mechanisms and causes? We answer these questions by deriving and thoroughly analyzing an analytically tractable theory of SSL loss landscapes. In this theory, we identify the causes of the dimensional collapse and study the effect of normalization and bias. Finally, we leverage the interpretability afforded by the analytical theory to understand how dimensional collapse can be beneficial and what affects the robustness of SSL against data imbalance.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 2 Oct 2022.

Builds on

44 of 51 references.

References · 44

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
32 more
  1. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  2. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  5. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  6. 2020Understanding the Behaviour of Contrastive Losscited by 397
  7. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  8. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  9. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  10. 2020Whitening for Self-Supervised Representation Learningcited by 278
  11. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  12. 2016Deep Learning without Poor Local Minimacited by 233
  13. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  14. 2020Representation Learning via Invariant Causal Mechanismscited by 197
  15. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  16. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  17. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  18. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  19. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
  20. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  21. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  22. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  23. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  24. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  25. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  26. 2021Self-supervised Representation Learning with Relative Predictive Codingcited by 27
  27. 2022Exact Solutions of a Deep Linear Networkcited by 24
  28. 2022Contrasting the landscape of contrastive and non-contrastive learningcited by 24
  29. 2022Posterior Collapse of a Linear Latent Variable Modelcited by 18
  30. 2022Toward a Geometrical Understanding of Self-supervised Contrastive Learningcited by 17
  31. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
  32. 2022Exact Phase Transitions in Deep Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 8; 2025: 2; 2026: 1.

19 papers.

Sort:

  1. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  2. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  3. 2025Symmetry in Neural Network Parameter Spacescited by 31
  4. 2023LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architecturescited by 27
  5. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  6. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  7. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  8. 2023Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descentcited by 8
  9. 2024Remove Symmetries to Control Model Expressivity and Improve Optimizationcited by 8
  10. 2023Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapsecited by 6
  11. 2024Infinite Width Limits of Self Supervised Neural Networkscited by 4
  12. 2023Representation Learning Dynamics of Self-Supervised Modelscited by 3
7 more
  1. 2025Dual Perspectives on Non-Contrastive Self-Supervised Learningcited by 3
  2. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
  3. 2024PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesiscited by 2
  4. 2024On Learning Discriminative Features from Synthesized Data for Self-Supervised Fine-Grained Visual Recognitioncited by 2
  5. 2024When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?cited by 1
  6. 2024Training Dynamics of Nonlinear Contrastive Learning Model in the High Dimensional Limitcited by 1
  7. 2026Asymptotic and Finite-Time Guarantees for Langevin-Based Temperature Annealing in InfoNCEcited by 1