# NeuralEF: Deconstructing Kernels by Deep Neural Networks

arXiv: https://arxiv.org/abs/2205.00165 · HTML: https://mlsanity.com/paper/2205.00165/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 30 Apr 2022.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://mlsanity.com/paper/1810.05148/) (2018)
- [Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels](https://mlsanity.com/paper/1905.13192/) (2019)
- [Neural Tangents: Fast and Easy Infinite Neural Networks in Python](https://mlsanity.com/paper/1912.02803/) (2019)
- [Deep Convolutional Networks as shallow Gaussian Processes](https://mlsanity.com/paper/1808.05587/) (2018)
- [Orthogonal Random Features](https://mlsanity.com/paper/1610.09072/) (2016)
- [Approximate Inference Turns Deep Networks into Gaussian Processes](https://mlsanity.com/paper/1906.01930/) (2019)
- ['In-Between' Uncertainty in Bayesian Neural Networks](https://mlsanity.com/paper/1906.11537/) (2019)
- [EigenGame: PCA as a Nash Equilibrium](https://mlsanity.com/paper/2010.00554/) (2020)
- [Spectral Inference Networks: Unifying Deep and Spectral Learning](https://mlsanity.com/paper/1806.02215/) (2018)
- [Quadrature-based features for kernel approximation](https://mlsanity.com/paper/1802.03832/) (2018)

## Built on by (most cited first, 27 of 27)

- [Improved Operator Learning by Orthogonal Attention](https://mlsanity.com/paper/2310.12487/) (2023)
- [Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions](https://mlsanity.com/paper/2210.01883/) (2022)
- [Accelerated Linearized Laplace Approximation for Bayesian Deep Learning](https://mlsanity.com/paper/2210.12642/) (2022)
- [Neural Eigenfunctions Are Structured Representation Learners](https://mlsanity.com/paper/2210.12637/) (2022)
- [Operator SVD with Neural Networks via Nested Low-Rank Approximation](https://mlsanity.com/paper/2402.03655/) (2024)
- [A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspaces](https://mlsanity.com/paper/2212.04025/) (2022)
- [Harnessing small projectors and multiple views for efficient vision pretraining](https://mlsanity.com/paper/2312.10725/) (2023)
- [Efficient Parametric SVD of Koopman Operator for Stochastic Dynamical Systems](https://mlsanity.com/paper/2507.07222/) (2025)
- [Learning Neural Eigenfunctions for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2304.02841/) (2023)
- [Demystifying Spectral Feature Learning for Instrumental Variable Regression](https://mlsanity.com/paper/2506.10899/) (2025)
- [STNet: Spectral Transformation Network for Solving Operator Eigenvalue Problem](https://mlsanity.com/paper/2510.23986/) (2025)
- [Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel](https://mlsanity.com/paper/2205.07384/) (2022)
- [Representations and Exploration for Deep Reinforcement Learning using Singular Value Decomposition](https://mlsanity.com/paper/2305.00654/) (2023)
- [Generalizable Spectral Embedding with an Application to UMAP](https://mlsanity.com/paper/2501.11305/) (2025)
- [Accelerating Eigenvalue Dataset Generation via Chebyshev Subspace Filter](https://mlsanity.com/paper/2510.23215/) (2025)
- [Contrastive Learning as Kernel Approximation](https://mlsanity.com/paper/2309.02651/) (2023)
- [Compact Vision Transformer by Reduction of Kernel Complexity](https://mlsanity.com/paper/2507.12780/) (2025)
- [Revisiting Orbital Minimization Method for Neural Operator Decomposition](https://mlsanity.com/paper/2510.21952/) (2025)
- [Eigenfunction Extraction for Ordered Representation Learning](https://mlsanity.com/paper/2510.24672/) (2025)
- [Learning-Guided Integration Contours Construction for Fast Large-Scale Generalized Eigensolvers](https://mlsanity.com/paper/2511.01927/) (2025)
- [NeuraLSP: A Neural Spectral Preconditioner for Accelerating PDE Solvers](https://mlsanity.com/paper/2601.20174/) (2026)
- [From Complex Dynamics to DynFormer: Rethinking Transformers for PDEs](https://mlsanity.com/paper/2603.03112/) (2026)
- [Minimax Rates and Spectral Distillation for Tree Ensembles](https://mlsanity.com/paper/2605.11841/) (2026)
- [Learning Orthonormal Bases for Function Spaces](https://mlsanity.com/paper/2605.19959/) (2026)
- [Score the Algebra, Not the Span: Dimension Reduction for Transfer Operator Models of Dynamical Systems](https://mlsanity.com/paper/2608.18918/) (2026)
- [Revisiting Spectral Representations in Generative Diffusion Models](https://mlsanity.com/paper/2609.08253/) (2026)
- [Low-Fidelity FDM Spectral Guidance for Neural Eigenvalue Solvers](https://mlsanity.com/paper/2610.04695/) (2026)

## Abstract (arXiv)

Learning the principal eigenfunctions of an integral operator defined by a kernel and a data distribution is at the core of many machine learning problems. Traditional nonparametric solutions based on the Nystr"om formula suffer from scalability issues. Recent work has resorted to a parametric approach, i.e., training neural networks to approximate the eigenfunctions. However, the existing method relies on an expensive orthogonalization step and is difficult to implement. We show that these problems can be fixed by using a new series of objective functions that generalizes the EigenGame~citepgemp2020eigengame to function space. We test our method on a variety of supervised and unsupervised learning problems and show it provides accurate approximations to the eigenfunctions of polynomial, radial basis, neural network Gaussian process, and neural tangent kernels. Finally, we demonstrate our method can scale up linearised Laplace approximation of deep neural networks to modern image classification datasets through approximating the Gauss-Newton matrix. Code is available at urlhttps://github.com/thudzj/neuraleigenfunction.