# Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nystr"om Method, and Use of Kernels in Machine Learning: Tutorial and Survey

arXiv: https://arxiv.org/abs/2106.08443 · HTML: https://mlsanity.com/paper/2106.08443/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 159 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [Kernel Mean Embedding of Distributions: A Review and Beyond](https://mlsanity.com/paper/1605.09522/) (2016)
- [Generative Moment Matching Networks](https://mlsanity.com/paper/1502.02761/) (2015)
- [Less is More: Nystr"om Computational Regularization](https://mlsanity.com/paper/1507.04717/) (2015)
- [Eigenvalue and Generalized Eigenvalue Problems: Tutorial](https://mlsanity.com/paper/1903.11240/) (2019)
- [Kernel Distribution Embeddings: Universal Kernels, Characteristic Kernels and Kernel Metrics on Distributions](https://mlsanity.com/paper/1604.05251/) (2016)
- [Metrizing Weak Convergence with Maximum Mean Discrepancies](https://mlsanity.com/paper/2006.09268/) (2020)
- [Feature Selection and Feature Extraction in Pattern Analysis: A Literature Review](https://mlsanity.com/paper/1905.02845/) (2019)
- [Locally Linear Embedding and its Variants: Tutorial and Survey](https://mlsanity.com/paper/2011.10925/) (2020)
- [Multidimensional Scaling, Sammon Mapping, and Isomap: Tutorial and Survey](https://mlsanity.com/paper/2009.08136/) (2020)
- [Unsupervised and Supervised Principal Component Analysis: Tutorial](https://mlsanity.com/paper/1906.03148/) (2019)
- [Fisher and Kernel Fisher Discriminant Analysis: Tutorial](https://mlsanity.com/paper/1906.09436/) (2019)
- [Advanced Stationary and Non-Stationary Kernel Designs for Domain-Aware Gaussian Processes](https://mlsanity.com/paper/2102.03432/) (2021)
- [Greedy Column Subset Selection for Large-scale Data Sets](https://mlsanity.com/paper/1312.6838/) (2013)
- [A Summary Of The Kernel Matrix, And How To Learn It Effectively Using Semidefinite Programming](https://mlsanity.com/paper/1709.06557/) (2017)
- [Theoretical Insights into the Use of Structural Similarity Index In Generative Models and Inferential Autoencoders](https://mlsanity.com/paper/2004.01864/) (2020)

## Built on by (most cited first, 30 of 37)

- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning](https://mlsanity.com/paper/2206.01342/) (2022)
- [Taming graph kernels with random features](https://mlsanity.com/paper/2305.00156/) (2023)
- [KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey](https://mlsanity.com/paper/2110.01858/) (2021)
- [ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models](https://mlsanity.com/paper/2501.11175/) (2025)
- [Spectral, Probabilistic, and Deep Metric Learning: Tutorial and Survey](https://mlsanity.com/paper/2201.09267/) (2022)
- [Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspective](https://mlsanity.com/paper/2311.16646/) (2023)
- [Learning of Population Dynamics: Inverse Optimization Meets JKO Scheme](https://mlsanity.com/paper/2506.01502/) (2025)
- [Johnson-Lindenstrauss Lemma, Linear and Nonlinear Random Projections, Random Fourier Features, and Random Kitchen Sinks: Tutorial and Survey](https://mlsanity.com/paper/2108.04172/) (2021)
- [Contrastive Corpus Attribution for Explaining Representations](https://mlsanity.com/paper/2210.00107/) (2022)
- [An Exact Kernel Equivalence for Finite Classification Models](https://mlsanity.com/paper/2308.00824/) (2023)
- [Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey](https://mlsanity.com/paper/2111.13282/) (2021)
- [Pseudo-keypoint RKHS Learning for Self-supervised 6DoF Pose Estimation](https://mlsanity.com/paper/2311.09500/) (2023)
- [Attention as Robust Representation for Time Series Forecasting](https://mlsanity.com/paper/2402.05370/) (2024)
- [On Foundation Models for Dynamical Systems from Purely Synthetic Data](https://mlsanity.com/paper/2412.00395/) (2024)
- [A Unified Theory of Sinusoidal Activation Families for Implicit Neural Representations](https://mlsanity.com/paper/2502.00869/) (2025)
- [On Mitigating the Utility-Loss in Differentially Private Learning: A new Perspective by a Geometrically Inspired Kernel Approach](https://mlsanity.com/paper/2304.01300/) (2023)
- [Learning to Embed Distributions via Maximum Kernel Entropy](https://mlsanity.com/paper/2408.00549/) (2024)
- [Convolutional Filtering with RKHS Algebras](https://mlsanity.com/paper/2411.01341/) (2024)
- [Tensor Network Based Feature Learning Model](https://mlsanity.com/paper/2512.02547/) (2025)
- [Parameterized Quantum Circuits with Quantum Kernels for Machine Learning: A Hybrid Quantum-Classical Approach](https://mlsanity.com/paper/2209.14449/) (2022)
- [Model Reprogramming Demystified: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2506.00620/) (2025)
- [Escaping Plato's Cave: JAM for Aligning Independently Trained Vision and Language Models](https://mlsanity.com/paper/2507.01201/) (2025)
- [Linearization Explains Fine-Tuning in Large Language Models](https://mlsanity.com/paper/2602.08239/) (2026)
- [Unified Framework for Spectral Dimensionality Reduction, Maximum Variance Unfolding, and Kernel Learning By Semidefinite Programming: Tutorial and Survey](https://mlsanity.com/paper/2106.15379/) (2021)
- [Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey](https://mlsanity.com/paper/2110.09620/) (2021)
- [Barron Space for Graph Convolution Neural Networks](https://mlsanity.com/paper/2311.02838/) (2023)
- [A Distributional Treatment of Real2Sim2Real for Object-Centric Agent Adaptation in Vision-Driven Deformable Linear Object Manipulation](https://mlsanity.com/paper/2502.18615/) (2025)
- [STRIDE: Subset-Free Functional Decomposition for XAI in Tabular Settings](https://mlsanity.com/paper/2509.09070/) (2025)
- [Sketch-Augmented Features Improve Learning Long-Range Dependencies in Graph Neural Networks](https://mlsanity.com/paper/2511.03824/) (2025)

## Abstract (arXiv)

This is a tutorial and survey paper on kernels, kernel methods, and related fields. We start with reviewing the history of kernels in functional analysis and machine learning. Then, Mercer kernel, Hilbert and Banach spaces, Reproducing Kernel Hilbert Space (RKHS), Mercer's theorem and its proof, frequently used kernels, kernel construction from distance metric, important classes of kernels (including bounded, integrally positive definite, universal, stationary, and characteristic kernels), kernel centering and normalization, and eigenfunctions are explained in detail. Then, we introduce types of use of kernels in machine learning including kernel methods (such as kernel support vector machines), kernel learning by semi-definite programming, Hilbert-Schmidt independence criterion, maximum mean discrepancy, kernel mean embedding, and kernel dimensionality reduction. We also cover rank and factorization of kernel matrix as well as the approximation of eigenfunctions and kernels using the Nystr"om method. This paper can be useful for various fields of science including machine learning, dimensionality reduction, functional analysis in mathematics, and mathematical physics in quantum mechanics.