mlsanityData to 14 Oct 2026

Reverse Engineering Self-Supervised Learning

arXiv 2305.15614 ↗First posted 24 May 2023cs.LG cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) is a powerful tool in machine learning, but understanding the learned representations and their underlying mechanisms remains a challenge. This paper presents an in-depth empirical analysis of SSL-trained representations, encompassing diverse models, architectures, and hyperparameters. Our study reveals an intriguing aspect of the SSL training process: it inherently facilitates the clustering of samples with respect to semantic labels, which is surprisingly driven by the SSL objective's regularization term. This clustering process not only enhances downstream classification but also compresses the data information. Furthermore, we establish that SSL-trained representations align more closely with semantic classes rather than random classes. Remarkably, we show that learned representations align with semantic classes across various hierarchical levels, and this alignment increases during training and when moving deeper into the network. Our findings provide valuable insights into SSL's representation learning mechanisms and their impact on performance across different sets of classes.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 50 of its 69 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

50 of 69 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
38 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  6. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  7. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  8. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  9. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  10. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  11. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  12. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  13. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  14. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  15. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  16. 2016Learning Representations for Automatic Colorizationcited by 251
  17. 2023A Cookbook of Self-Supervised Learningcited by 250
  18. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  19. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  20. 2018Large Margin Deep Networks for Classificationcited by 112
  21. 2020Parametric UMAP embeddings for representation and semi-supervised learningcited by 104
  22. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  23. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  24. 2023What Do Self-Supervised Vision Transformers Learn?cited by 93
  25. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  26. 2020OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learningcited by 65
  27. 2016Convolutional Neural Networks Analyzed via Convolutional Sparse Codingcited by 52
  28. 2020Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Taskscited by 47
  29. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  30. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 34
  31. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  32. 2022Information Flow in Deep Neural Networkscited by 22
  33. 2022Nearest Class-Center Simplification through Intermediate Layerscited by 20
  34. 2018DNN or k-NN: That is the Generalize vs. Memorize Questioncited by 16
  35. 2022What Do We Maximize in Self-Supervised Learning?cited by 15
  36. 2019Information in Infinite Ensembles of Infinitely-Wide Neural Networkscited by 14
  37. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  38. 2021Sparsity-Probe: Analysis tool for Deep Learning Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 15; 2025: 6; 2026: 8.

33 papers.

Sort:

By area:

  1. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
  2. 2024EEG2Rep: Enhancing Self-supervised EEG Representation Through Informative Masked Inputscited by 50
  3. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Modelscited by 34
  4. 2024The Entropy Enigma: Success and Failure of Entropy Minimizationcited by 24
  5. 2024DetailCLIP: Detail-Oriented CLIP for Fine-Grained Taskscited by 15
  6. 2023Variance-Covariance Regularization Improves Representation Learningcited by 14
  7. 2024P2OT: Progressive Partial Optimal Transport for Deep Imbalanced Clusteringcited by 13
  8. 2024Words in Motion: Extracting Interpretable Control Vectors for Motion Transformerscited by 9
  9. 2025Register and [CLS] tokens yield a decoupling of local and global features in large ViTscited by 9
  10. 2025Clustering Properties of Self-Supervised Learningcited by 8
  11. 2024Learning to Compress: Local Rank and Information Compression in Deep Neural Networkscited by 7
  12. 2024HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithmscited by 7
21 more
  1. 2023KERMIT: Knowledge Graph Completion of Enhanced Relation Modeling with Inverse Transformationcited by 6
  2. 2025BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principlecited by 6
  3. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  4. 2024A Probabilistic Model Behind Self-Supervised Learningcited by 5
  5. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learningcited by 5
  6. 2024Frequency-Guided Masking for Enhanced Vision Self-Supervised Learningcited by 4
  7. 2026Understanding Self-Supervised Learning via Latent Distribution Matchingcited by 4
  8. 2024Self-Supervised Learning for Text Recognition: A Critical Surveycited by 3
  9. 2026Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learningcited by 3
  10. 2024SP2OT: Semantic-Regularized Progressive Partial Optimal Transport for Imbalanced Clusteringcited by 2
  11. 2025On the Alignment Between Supervised and Self-Supervised Contrastive Learningcited by 2
  12. 2024Cross-Level Multi-Instance Distillation for Self-Supervised Fine-Grained Visual Categorizationcited by 1
  13. 2026Learning Hierarchical Knowledge in Text-Rich Networks with Taxonomy-Informed Representation Learningcited by 1
  14. 2023Splicing Up Your Predictions with RNA Contrastive Learningcited by 0
  15. 2024Real-world Instance-specific Image Goal Navigation: Bridging Domain Gaps via Contrastive Learningcited by 0
  16. 2024Deep Sparse Latent Feature Models for Knowledge Graph Completioncited by 0
  17. 2026Information Theoretic Perspective on Representation Learningcited by 0
  18. 2026Radial Basis Function Networks as Projection Heads in Self-Supervised Learningcited by 0
  19. 2026Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Modelscited by 0
  20. 2026Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisancecited by 0
  21. 2026Which Tasks Survive Self-Supervised Learning?cited by 0