# Kernel Flows: from learning kernels from data into the abyss

arXiv: https://arxiv.org/abs/1808.04475 · HTML: https://mlsanity.com/paper/1808.04475/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2018.

## Built on by (most cited first, 30 of 49)

- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [Solving and Learning Nonlinear PDEs with Gaussian Processes](https://mlsanity.com/paper/2103.12959/) (2021)
- [Kernel Methods are Competitive for Operator Learning](https://mlsanity.com/paper/2304.13202/) (2023)
- [Sparse Cholesky Factorization for Solving Nonlinear PDEs via Gaussian Processes](https://mlsanity.com/paper/2304.01294/) (2023)
- [Computational Graph Completion](https://mlsanity.com/paper/2110.10323/) (2021)
- [Physics-Informed Computer Vision: A Review and Perspectives](https://mlsanity.com/paper/2305.18035/) (2023)
- [Distribution learning via neural differential equations: a nonparametric statistical perspective](https://mlsanity.com/paper/2309.01043/) (2023)
- [Error Analysis of Kernel/GP Methods for Nonlinear and Parametric PDEs](https://mlsanity.com/paper/2305.04962/) (2023)
- [Learning to Forecast Dynamical Systems from Streaming Data](https://mlsanity.com/paper/2109.09703/) (2021)
- [Data-driven geophysical forecasting: Simple, low-cost, and accurate baselines with kernel methods](https://mlsanity.com/paper/2103.10935/) (2021)
- [One-Shot Learning of Stochastic Differential Equations with Data Adapted Kernels](https://mlsanity.com/paper/2209.12086/) (2022)
- [Learning Dynamical Systems from Data: A Simple Cross-Validation Perspective, Part V: Sparse Kernel Flows for 132 Chaotic Dynamical Systems](https://mlsanity.com/paper/2301.10321/) (2023)
- [Learning dynamical systems from data: A simple cross-validation perspective, part III: Irregularly-Sampled Time Series](https://mlsanity.com/paper/2111.13037/) (2021)
- [Subspace Decomposition based DNN algorithm for elliptic type multi-scale PDEs](https://mlsanity.com/paper/2112.06660/) (2021)
- [Nonparametric learning of kernels in nonlocal operators](https://mlsanity.com/paper/2205.11006/) (2022)
- [Learning "best" kernels from data in Gaussian process regression. With application to aerodynamics](https://mlsanity.com/paper/2206.02563/) (2022)
- [Data adaptive RKHS Tikhonov regularization for learning kernels in operators](https://mlsanity.com/paper/2203.03791/) (2022)
- [A Kernel Approach for PDE Discovery and Operator Learning](https://mlsanity.com/paper/2210.08140/) (2022)
- [Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features](https://mlsanity.com/paper/2407.00584/) (2024)
- [Operator Learning at Machine Precision](https://mlsanity.com/paper/2511.19980/) (2025)
- [Symmetric and antisymmetric kernels for machine learning problems in quantum physics and chemistry](https://mlsanity.com/paper/2103.17233/) (2021)
- [Kernel Methods for the Approximation of the Eigenfunctions of the Koopman Operator](https://mlsanity.com/paper/2412.16588/) (2024)
- [Gaussian Process Hydrodynamics](https://mlsanity.com/paper/2209.10707/) (2022)
- [Multi-fidelity Gaussian process surrogate modeling for regression problems in physics](https://mlsanity.com/paper/2404.11965/) (2024)
- [Kernel Sum of Squares for Data Adapted Kernel Learning of Dynamical Systems from Data: A global optimization approach](https://mlsanity.com/paper/2408.06465/) (2024)
- [Self-test loss functions for learning weak-form operators and gradient flows](https://mlsanity.com/paper/2412.03506/) (2024)
- [Solving Maxwell's Equation in 2D with Neural Networks with Local Converging Inputs](https://mlsanity.com/paper/2302.02860/) (2023)
- [Bridging Algorithmic Information Theory and Machine Learning: A New Approach to Kernel Learning](https://mlsanity.com/paper/2311.12624/) (2023)
- [Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes](https://mlsanity.com/paper/2510.05568/) (2025)
- [Analysis of Structured Deep Kernel Networks](https://mlsanity.com/paper/2105.07228/) (2021)

## Abstract (arXiv)

Learning can be seen as approximating an unknown function by interpolating the training data. Kriging offers a solution to this problem based on the prior specification of a kernel. We explore a numerical approximation approach to kernel selection/construction based on the simple premise that a kernel must be good if the number of interpolation points can be halved without significant loss in accuracy (measured using the intrinsic RKHS norm |·| associated with the kernel). We first test and motivate this idea on a simple problem of recovering the Green's function of an elliptic PDE (with inhomogeneous coefficients) from the sparse observation of one of its solutions. Next we consider the problem of learning non-parametric families of deep kernels of the form K1(Fn(x),Fn(x')) with Fn+1=(Id+ε Gn+1)circ Fn and Gn+1 ∈ operatornameSpanK1(Fn(xi),·). With the proposed approach constructing the kernel becomes equivalent to integrating a stochastic data driven dynamical system, which allows for the training of very deep (bottomless) networks and the exploration of their properties. These networks learn by constructing flow maps in the kernel and input spaces via incremental data-dependent deformations/perturbations (appearing as the cooperative counterpart of adversarial examples) and, at profound depths, they (1) can achieve accurate classification from only one data point per class (2) appear to learn archetypes of each class (3) expand distances between points that are in different classes and contract distances between points in the same class. For kernels parameterized by the weights of Convolutional Neural Networks, minimizing approximation errors incurred by halving random subsets of interpolation points, appears to outperform training (the same CNN architecture) with relative entropy and dropout.