mlsanityData to 14 Oct 2026

Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimization

arXiv 2509.18997 ↗First posted 23 Sep 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Representation learning from unlabeled data has been extensively studied in statistics, data science and signal processing with a rich literature on techniques for dimension reduction, compression, multi-dimensional scaling among others. However, current deep learning models use new principles for unsupervised representation learning that cannot be easily analyzed using classical theories. For example, visual foundation models have found tremendous success using self-supervision or denoising/masked autoencoders, which effectively learn representations from massive amounts of unlabeled data. However, it remains difficult to characterize the representations learned by these models and to explain why they perform well for diverse prediction tasks or show emergent behavior. To answer these questions, one needs to combine mathematical tools from statistics and optimization. This paper provides an overview of recent theoretical advances in representation learning from unlabeled data and mentions our contributions in this direction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 51 of its 81 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

51 of 81 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  9. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  10. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  12. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
38 more
  1. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  2. 2021Explaining Neural Scaling Lawscited by 363
  3. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  4. 2021Unsolved Problems in ML Safetycited by 292
  5. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  6. 2016Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clusteringcited by 237
  7. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  8. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  9. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  10. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  11. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  12. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  13. 2020Learning Parities with Neural Networkscited by 78
  14. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  15. 2024Learning by Reconstruction Produces Uninformative Features For Perceptioncited by 55
  16. 2019Loss Landscapes of Regularized Linear Autoencoderscited by 46
  17. 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 41
  18. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  19. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 40
  20. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
  21. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  22. 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
  23. 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
  24. 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
  25. 2020Regularized linear autoencoders recover the principal components, eventuallycited by 26
  26. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  27. 2023On the Provable Advantage of Unsupervised Pretrainingcited by 22
  28. 2019PAC-Bayesian Contrastive Unsupervised Representation Learningcited by 20
  29. 2021On the Equivalence between Neural Network and Support Vector Machinecited by 17
  30. 2023Double Descent and Overfitting under Noisy Inputs and Distribution Shift for Linear Denoiserscited by 14
  31. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 14
  32. 2023Non-Parametric Representation Learning with Kernelscited by 11
  33. 2024On the Generalization Ability of Unsupervised Pretrainingcited by 8
  34. 2024Exact Certification of (Graph) Neural Networks Against Label Poisoningcited by 6
  35. 2024Tight PAC-Bayesian Risk Certificates for Contrastive Learningcited by 6
  36. 2024A Theoretical Characterization of Optimal Data Augmentations in Self-Supervised Learningcited by 4
  37. 2024Infinite Width Limits of Self Supervised Neural Networkscited by 4
  38. 2022Improved Representation Learning Through Tensorized Autoencoderscited by 1

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.