# Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research

arXiv: https://arxiv.org/abs/2504.13101 · HTML: https://mlsanity.com/paper/2504.13101/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 92 of its 136 references that are in the map.
- First posted on arXiv 17 Apr 2025.

## Builds on (most cited first, 30 of 92)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Gradient-Based Neural DAG Learning](https://mlsanity.com/paper/1906.02226/) (2019)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [SGD on Neural Networks Learns Functions of Increasing Complexity](https://mlsanity.com/paper/1905.11604/) (2019)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)

## Built on by (most cited first, 9 of 9)

- [Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning](https://mlsanity.com/paper/2505.12477/) (2025)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Understanding Self-Supervised Learning via Latent Distribution Matching](https://mlsanity.com/paper/2605.03517/) (2026)
- [The Observer Effect in World Models: Invasive Adaptation Corrupts Latent Physics](https://mlsanity.com/paper/2602.12218/) (2026)
- [Self-Supervised Learning from Structural Invariance](https://mlsanity.com/paper/2602.02381/) (2026)
- [The Impact of Semantic Pairs on Self-Supervised Representation Learning](https://mlsanity.com/paper/2510.08722/) (2025)
- [Eigenfunction Extraction for Ordered Representation Learning](https://mlsanity.com/paper/2510.24672/) (2025)
- [Similarity of Neural Network Representations in Superposition](https://mlsanity.com/paper/2604.00208/) (2026)
- [An Investigation of the NeurIPS and ICML 2025 Position Tracks](https://mlsanity.com/paper/2608.16894/) (2026)

## Abstract (arXiv)

Self-Supervised Learning (SSL) powers many current AI systems. As research interest and investment grow, the SSL design space continues to expand. The Platonic view of SSL, following the Platonic Representation Hypothesis (PRH), suggests that despite different methods and engineering approaches, all representations converge to the same Platonic ideal. However, this phenomenon lacks precise theoretical explanation. By synthesizing evidence from Identifiability Theory (IT), we show that the PRH can emerge in SSL. However, current IT cannot explain SSL's empirical success. To bridge the gap between theory and practice, we propose expanding IT into what we term Singular Identifiability Theory (SITh), a broader theoretical framework encompassing the entire SSL pipeline. SITh would allow deeper insights into the implicit data assumptions in SSL and advance the field towards learning more interpretable and generalizable representations. We highlight three critical directions for future research: 1) training dynamics and convergence properties of SSL; 2) the impact of finite samples, batch size, and data diversity; and 3) the role of inductive biases in architecture, augmentations, initialization schemes, and optimizers.