mlsanityData to 14 Oct 2026

Joint Embedding Self-Supervised Learning in the Kernel Regime

arXiv 2209.14884 ↗First posted 29 Sep 2022cs.LG cs.AI stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

The fundamental goal of self-supervised learning (SSL) is to produce useful representations of data without access to any labels for classifying the data. Modern methods in SSL, which form representations based on known or constructed relationships between samples, have been particularly effective at this task. Here, we aim to extend this framework to incorporate algorithms based on kernel methods where embeddings are constructed by linear maps acting on the feature space of a kernel. In this kernel regime, we derive methods to find the optimal form of the output representations for contrastive and non-contrastive loss functions. This procedure produces a new representation space with an inner product denoted as the induced kernel which generally correlates points which are related by an augmentation in kernel space and de-correlates points otherwise. We analyze our kernel model on small datasets to identify common features of self-supervised learning algorithms and gain theoretical insights into their performance on downstream tasks.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 30 of its 54 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

30 of 54 references.

References · 30

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  8. 2021Convex optimizationcited by 2,515
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  11. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  12. 2017Deep Neural Networks as Gaussian Processescited by 622
18 more
  1. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  2. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  3. 2020Power of data in quantum machine learningcited by 338
  4. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  5. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  6. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  7. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 232
  8. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 180
  9. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  10. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  11. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  12. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  13. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  14. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  15. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  16. 2013A Survey on Metric Learning for Feature Vectors and Structured Datacited by 58
  17. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  18. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 6.

12 papers.

  1. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  2. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  3. 2023Non-Parametric Representation Learning with Kernelscited by 11
  4. 2023Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Needcited by 9
  5. 2024Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Lossescited by 7
  6. 2023Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysiscited by 6
  7. 2024A Theoretical Characterization of Optimal Data Augmentations in Self-Supervised Learningcited by 4
  8. 2024Infinite Width Limits of Self Supervised Neural Networkscited by 4
  9. 2023Representation Learning Dynamics of Self-Supervised Modelscited by 3
  10. 2024GPS-SSL: Guided Positive Sampling to Inject Prior Into Self-Supervised Learningcited by 3
  11. 2024Navigating the Effect of Parametrization for Dimensionality Reductioncited by 3
  12. 2024When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?cited by 1