# Function Spaces Without Kernels: Learning Compact Hilbert Space Representations

arXiv: https://arxiv.org/abs/2509.20605 · HTML: https://mlsanity.com/paper/2509.20605/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 24 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 8 of 8)

- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition](https://mlsanity.com/paper/1710.04340/) (2017)
- [Fastfood: Approximate Kernel Expansions in Loglinear Time](https://mlsanity.com/paper/1408.3060/) (2014)
- [Structure-preserving Sparse Identification of Nonlinear Dynamics for Data-driven Modeling](https://mlsanity.com/paper/2109.05364/) (2021)
- [Zero-Shot Reinforcement Learning via Function Encoders](https://mlsanity.com/paper/2401.17173/) (2024)
- [Zero-Shot Transfer of Neural ODEs](https://mlsanity.com/paper/2405.08954/) (2024)
- [Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces](https://mlsanity.com/paper/2501.18373/) (2025)
- [Online Adaptation of Terrain-Aware Dynamics for Planning in Unstructured Environments](https://mlsanity.com/paper/2506.04484/) (2025)

## Built on by (most cited first, 1 of 1)

- [Zero-Shot Function Encoder-Based Differentiable Predictive Control](https://mlsanity.com/paper/2511.05757/) (2025)

## Abstract (arXiv)

Function encoders are a recent technique that learn neural network basis functions to form compact, adaptive representations of Hilbert spaces of functions. We show that function encoders provide a principled connection to feature learning and kernel methods by defining a kernel through an inner product of the learned feature map. This kernel-theoretic perspective explains their ability to scale independently of dataset size while adapting to the intrinsic structure of data, and it enables kernel-style analysis of neural models. Building on this foundation, we develop two training algorithms that learn compact bases: a progressive training approach that constructively grows bases, and a train-then-prune approach that offers a computationally efficient alternative after training. Both approaches use principles from PCA to reveal the intrinsic dimension of the learned space. In parallel, we derive finite-sample generalization bounds using Rademacher complexity and PAC-Bayes techniques, providing inference time guarantees. We validate our approach on a polynomial benchmark with a known intrinsic dimension, and on nonlinear dynamical systems including a Van der Pol oscillator and a two-body orbital model, demonstrating that the same accuracy can be achieved with substantially fewer basis functions. This work suggests a path toward neural predictors with kernel-level guarantees, enabling adaptable models that are both efficient and principled at scale.