Representation Learning Dynamics of Self-Supervised Models
Not on the map. Why?
Abstract (arXiv)
Self-Supervised Learning (SSL) is an important paradigm for learning representations from unlabelled data, and SSL with neural networks has been highly successful in practice. However current theoretical analysis of SSL is mostly restricted to generalisation error bounds. In contrast, learning dynamics often provide a precise characterisation of the behaviour of neural networks based models but, so far, are mainly known in supervised settings. In this paper, we study the learning dynamics of SSL models, specifically representations obtained by minimising contrastive and non-contrastive losses. We show that a naive extension of the dymanics of multivariate regression to SSL leads to learning trivial scalar representations that demonstrates dimension collapse in SSL. Consequently, we formulate SSL objectives with orthogonality constraints on the weights, and derive the exact (network width independent) learning dynamics of the SSL models trained using gradient descent on the Grassmannian manifold. We also argue that the infinite width approximation of SSL models significantly deviate from the neural tangent kernel approximations of supervised models. We numerically illustrate the validity of our theoretical findings, and discuss how the presented results provide a framework for further theoretical analysis of contrastive and non-contrastive SSL.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 62 of its 73 references that are in the map. First posted on arXiv 5 Sep 2023.
Builds on
62 of 73 references.
References · 62
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
38 more
- 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
- 2017Exploring Generalization in Deep Learningcited by 595
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
- 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
- 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
- 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
- 2020Whitening for Self-Supervised Representation Learningcited by 278
- 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
- 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 160
- 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
- 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
- 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
- 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
- 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
- 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
- 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
- 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 71
- 2020The Recurrent Neural Tangent Kernelcited by 67
- 2018Towards Understanding Regularization in Batch Normalizationcited by 56
- 2019Loss Landscapes of Regularized Linear Autoencoderscited by 45
- 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 40
- 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
- 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
- 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
- 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
- 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
- 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
- 2018Self-supervised Learning of Geometrically Stable Features Through Probabilistic Introspectioncited by 27
- 2021Max-Margin Contrastive Learningcited by 27
- 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
- 2022Exact Solutions of a Deep Linear Networkcited by 24
- 2022Contrasting the landscape of contrastive and non-contrastive learningcited by 24
- 2016WarpNet: Weakly Supervised Matching for Single-view Reconstructioncited by 22
50 most cited of 62 All 62 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.