# Non-Parametric Representation Learning with Kernels

arXiv: https://arxiv.org/abs/2309.02028 · HTML: https://mlsanity.com/paper/2309.02028/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 72 references that are in the map.
- First posted on arXiv 5 Sep 2023.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering](https://mlsanity.com/paper/1610.04794/) (2016)
- [A vector-contraction inequality for Rademacher complexities](https://mlsanity.com/paper/1605.00251/) (2016)
- [On the linearity of large non-linear models: when and why the tangent kernel is constant](https://mlsanity.com/paper/2010.01092/) (2020)
- [Limitations of Lazy Training of Two-layers Neural Networks](https://mlsanity.com/paper/1906.08899/) (2019)
- [Deep Equals Shallow for ReLU Networks in Kernel Regimes](https://mlsanity.com/paper/2009.14397/) (2020)
- [A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond](https://mlsanity.com/paper/2208.00173/) (2022)
- [Loss Landscapes of Regularized Linear Autoencoders](https://mlsanity.com/paper/1901.08168/) (2019)
- [Overparameterized Neural Networks Implement Associative Memory](https://mlsanity.com/paper/1909.12362/) (2019)
- [Learning with SGD and Random Features](https://mlsanity.com/paper/1807.06343/) (2018)
- [Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions](https://mlsanity.com/paper/2210.01883/) (2022)
- [The dynamics of representation learning in shallow, non-linear autoencoders](https://mlsanity.com/paper/2201.02115/) (2022)
- [WarpNet: Weakly Supervised Matching for Single-view Reconstruction](https://mlsanity.com/paper/1604.05592/) (2016)
- [Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression](https://mlsanity.com/paper/2306.00788/) (2023)
- [Learning Dynamics of Linear Denoising Autoencoders](https://mlsanity.com/paper/1806.05413/) (2018)
- [Benefits of Jointly Training Autoencoders: An Improved Neural Tangent Kernel Analysis](https://mlsanity.com/paper/1911.11983/) (2019)
- [Joint Embedding Self-Supervised Learning in the Kernel Regime](https://mlsanity.com/paper/2209.14884/) (2022)
- [Minimax Lower Bounds for Realizable Transductive Classification](https://mlsanity.com/paper/1602.03027/) (2016)
- [Bounds for Vector-Valued Function Estimation](https://mlsanity.com/paper/1606.01487/) (2016)

## Built on by (most cited first, 11 of 11)

- [Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens](https://mlsanity.com/paper/2310.13220/) (2023)
- [A Theoretical Characterization of Optimal Data Augmentations in Self-Supervised Learning](https://mlsanity.com/paper/2411.01767/) (2024)
- [Infinite Width Limits of Self Supervised Neural Networks](https://mlsanity.com/paper/2411.11176/) (2024)
- [Modeling Hierarchical Spaces: A Review and Unified Framework for Surrogate-Based Architecture Design](https://mlsanity.com/paper/2506.22621/) (2025)
- [Leveraging semantic similarity for experimentation with AI-generated treatments](https://mlsanity.com/paper/2510.21119/) (2025)
- [When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?](https://mlsanity.com/paper/2403.08673/) (2024)
- [Interpretable Self-Supervised Learning via Representer Landmarks and Nystr"om Approximation](https://mlsanity.com/paper/2509.24467/) (2025)
- [Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimization](https://mlsanity.com/paper/2509.18997/) (2025)
- [A Unified Non-Parametric and Interpretable Point Cloud Analysis via t-FCW Graph Representation](https://mlsanity.com/paper/2605.15475/) (2026)
- [Dual-Manifold Geometry Guided Representation Learning: Adaptive Coupling between Kernel and Data Spaces](https://mlsanity.com/paper/2608.12737/) (2026)
- [An Analysis of Self-supervised Pre-training with Dependent Samples](https://mlsanity.com/paper/2609.05031/) (2026)

## Abstract (arXiv)

Unsupervised and self-supervised representation learning has become popular in recent years for learning useful features from unlabelled data. Representation learning has been mostly developed in the neural network literature, and other models for representation learning are surprisingly unexplored. In this work, we introduce and analyze several kernel-based representation learning approaches: Firstly, we define two kernel Self-Supervised Learning (SSL) models using contrastive loss functions and secondly, a Kernel Autoencoder (AE) model based on the idea of embedding and reconstructing data. We argue that the classical representer theorems for supervised kernel machines are not always applicable for (self-supervised) representation learning, and present new representer theorems, which show that the representations learned by our kernel models can be expressed in terms of kernel matrices. We further derive generalisation error bounds for representation learning with kernel SSL and AE, and empirically evaluate the performance of these methods in both small data regimes as well as in comparison with neural network based models.