mlsanityData to 7 Oct 2026

Understanding Self-Supervised Learning via Latent Distribution Matching

arXiv 2605.03517 ↗First posted 5 May 2026cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) excels at finding general-purpose latent representations from complex data, yet lacks a unifying theoretical framework that explains the diverse existing methods and guides the design of new ones. We cast SSL as latent distribution matching (LDM): learning representations that maximize their log-probability under an assumed latent model (alignment), while maximizing latent entropy to prevent collapse (uniformity). This view unifies independent component analysis with contrastive, non-contrastive, and predictive SSL methods, including stop gradient approaches. Leveraging LDM, we derive a nonlinear, sampling-free Bayesian filtering model with a Kalman-based predictor for high-dimensional timeseries. We further prove that predictive LDM yields identifiable latent representations under mild assumptions, even with nonlinear predictors. Overall, LDM clarifies the assumptions behind established SSL methods and provides principled guidance for developing new approaches.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

52 of 72 references.

References · 52

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  8. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  11. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  12. 2019On Variational Bounds of Mutual Informationcited by 550
38 more
  1. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  2. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  4. 2019On Mutual Information Maximization for Representation Learningcited by 315
  5. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  6. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  7. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
  8. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  9. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  10. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  11. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  12. 2020Flows for simultaneous manifold learning and density estimationcited by 117
  13. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  14. 2022Learnable latent embeddings for joint behavioral and neural analysiscited by 113
  15. 2020On Linear Identifiability of Learned Representationscited by 103
  16. 2020ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICAcited by 102
  17. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  18. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  19. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 57
  20. 2023Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligencecited by 52
  21. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  22. 2023Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learningcited by 48
  23. 2023Reverse Engineering Self-Supervised Learningcited by 33
  24. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 30
  25. 2023Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputscited by 28
  26. 2022Representation Uncertainty in Self-Supervised Learning as Variational Inferencecited by 22
  27. 2023Lifting Architectural Constraints of Injective Flowscited by 17
  28. 2022Implicit variance regularization in non-contrastive SSLcited by 15
  29. 2025I-Con: A Unifying Framework for Representation Learningcited by 15
  30. 2023The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learningcited by 13
  31. 2025Gaussian Embeddings: How JEPAs Secretly Learn Your Data Densitycited by 13
  32. 2021Understanding Neural Networks with Logarithm Determinant Entropy Estimatorcited by 10
  33. 2021InfoNCE is variational inference in a recognition parameterised modelcited by 10
  34. 2024Self-supervised contrastive learning performs non-linear system identificationcited by 9
  35. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  36. 2025Your contrastive learning problem is secretly a distribution alignment problemcited by 6
  37. 2022Unsupervised representation learning with recognition-parametrised probabilistic modelscited by 5
  38. 2024A Probabilistic Model Behind Self-Supervised Learningcited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Converge to Surprise: Evolutionary Self-supervised Image Clusteringcited by 0
  2. 2026Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisancecited by 0
  3. 2026Learning Commute-Time-Preserving World Models for Planningcited by 0
  4. 2026Shared Gaussianization: What Gaussian Regularizers Certify About Contrastive Learning, and What They Misscited by 0