mlsanityData to 7 Oct 2026

Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research

arXiv 2504.13101 ↗First posted 17 Apr 2025cs.LG cs.AI stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) powers many current AI systems. As research interest and investment grow, the SSL design space continues to expand. The Platonic view of SSL, following the Platonic Representation Hypothesis (PRH), suggests that despite different methods and engineering approaches, all representations converge to the same Platonic ideal. However, this phenomenon lacks precise theoretical explanation. By synthesizing evidence from Identifiability Theory (IT), we show that the PRH can emerge in SSL. However, current IT cannot explain SSL's empirical success. To bridge the gap between theory and practice, we propose expanding IT into what we term Singular Identifiability Theory (SITh), a broader theoretical framework encompassing the entire SSL pipeline. SITh would allow deeper insights into the implicit data assumptions in SSL and advance the field towards learning more interpretable and generalizable representations. We highlight three critical directions for future research: 1) training dynamics and convergence properties of SSL; 2) the impact of finite samples, batch size, and data diversity; and 3) the role of inductive biases in architecture, augmentations, initialization schemes, and optimizers.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 92 of its 136 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

92 of 136 references.

References · 92

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  9. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  10. 2021Scaling Vision Transformerscited by 993
  11. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  12. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
38 more
  1. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  2. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  3. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  5. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  6. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  7. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  8. 2018Towards a Definition of Disentangled Representationscited by 294
  9. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
  10. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 270
  11. 2021Scaling Laws for Transfercited by 265
  12. 2019Gradient-Based Neural DAG Learningcited by 227
  13. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  14. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  15. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  16. 2021Towards Causal Representation Learningcited by 207
  17. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  18. 2023Generalization in diffusion models arises from geometry-adaptive harmonic representationscited by 179
  19. 2022Relative representations enable zero-shot latent space communicationcited by 175
  20. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  21. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  22. 2022Weakly supervised causal representation learningcited by 140
  23. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  24. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  25. 2015Deep Convolutional Inverse Graphics Networkcited by 121
  26. 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 121
  27. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  28. 2022Learnable latent embeddings for joint behavioral and neural analysiscited by 113
  29. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  30. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 105
  31. 2020ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICAcited by 102
  32. 2021Independent mechanism analysis, a new concept?cited by 96
  33. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  34. 2021Visual Representation Learning Does Not Generalize Strongly Within the Same Domaincited by 60
  35. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  36. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  37. 2023Causal Component Analysiscited by 56
  38. 2023Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolationcited by 56

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 33
  2. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  3. 2026Understanding Self-Supervised Learning via Latent Distribution Matchingcited by 4
  4. 2026The Observer Effect in World Models: Invasive Adaptation Corrupts Latent Physicscited by 3
  5. 2026Self-Supervised Learning from Structural Invariancecited by 1
  6. 2025The Impact of Semantic Pairs on Self-Supervised Representation Learningcited by 0
  7. 2025Eigenfunction Extraction for Ordered Representation Learningcited by 0
  8. 2026Similarity of Neural Network Representations in Superpositioncited by 0
  9. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0