Spectal Harmonics: Bridging Spectral Embedding and Matrix Completion in Self-Supervised Learning
Not on the map. Why?
Abstract (arXiv)
Self-supervised methods received tremendous attention thanks to their seemingly heuristic approach to learning representations that respect the semantics of the data without any apparent supervision in the form of labels. A growing body of literature is already being published in an attempt to build a coherent and theoretically grounded understanding of the workings of a zoo of losses used in modern self-supervised representation learning methods. In this paper, we attempt to provide an understanding from the perspective of a Laplace operator and connect the inductive bias stemming from the augmentation process to a low-rank matrix completion problem. To this end, we leverage the results from low-rank matrix completion to provide theoretical analysis on the convergence of modern SSL methods and a key property that affects their downstream performance.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 17 of its 38 references that are in the map. First posted on arXiv 31 May 2023.
Builds on
17 of 38 references.
References · 17
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
- 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
- 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
- 2020Whitening for Self-Supervised Representation Learningcited by 279
- 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
- 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
- 2021Decoupled Contrastive Learningcited by 143
- 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
- 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
5 more
- 2018Spectral Inference Networks: Unifying Deep and Spectral Learningcited by 35
- 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
- 2013Incoherence-Optimal Matrix Completioncited by 27
- 2018Exact Reconstruction of Euclidean Distance Geometry Problem Using Low-rank Matrix Completioncited by 18
- 2023Unsupervised Embedding Quality Evaluationcited by 9
Built on by
Citing papers per year: 2024: 2.
2 papers.