# Understanding neural networks with reproducing kernel Banach spaces

arXiv: https://arxiv.org/abs/2109.09710 · HTML: https://mlsanity.com/paper/2109.09710/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 69 references that are in the map.
- First posted on arXiv 20 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Kernel Mean Embedding of Distributions: A Review and Beyond](https://mlsanity.com/paper/1605.09522/) (2016)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020)
- [Generalization Properties of Learning with Random Features](https://mlsanity.com/paper/1602.04474/) (2016)
- [Path-SGD: Path-Normalized Optimization in Deep Neural Networks](https://mlsanity.com/paper/1506.02617/) (2015)
- [A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case](https://mlsanity.com/paper/1910.01635/) (2019)
- [How do infinite width bounded norm networks look in function space?](https://mlsanity.com/paper/1902.05040/) (2019)
- [On the Power and Limitations of Random Features for Understanding Neural Networks](https://mlsanity.com/paper/1904.00687/) (2019)
- [Approximation spaces of deep neural networks](https://mlsanity.com/paper/1905.01208/) (2019)
- [On Reproducing Kernel Banach Spaces: Generic Definitions and Unified Framework of Constructions](https://mlsanity.com/paper/1901.01002/) (2019)
- [Sparsity of solutions for variational inverse problems with finite-dimensional data](https://mlsanity.com/paper/1809.05045/) (2018)
- [On Representer Theorems and Convex Regularization](https://mlsanity.com/paper/1806.09810/) (2018)
- [A unifying representer theorem for inverse problems and machine learning](https://mlsanity.com/paper/1903.00687/) (2019)

## Built on by (most cited first, 30 of 50)

- [Nonparametric regression using over-parameterized shallow ReLU neural networks](https://mlsanity.com/paper/2306.08321/) (2023)
- [Deep Learning Meets Sparse Regularization: A Signal Processing Perspective](https://mlsanity.com/paper/2301.09554/) (2023)
- [Ridges, Neural Networks, and the Radon Transform](https://mlsanity.com/paper/2203.02543/) (2022)
- [Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compression](https://mlsanity.com/paper/2305.16534/) (2023)
- [Duality for Neural Networks through Reproducing Kernel Banach Spaces](https://mlsanity.com/paper/2211.05020/) (2022)
- [Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regression](https://mlsanity.com/paper/2304.01561/) (2023)
- [Sparse Representer Theorems for Learning in Reproducing Kernel Banach Spaces](https://mlsanity.com/paper/2305.12584/) (2023)
- [ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Models](https://mlsanity.com/paper/2305.15598/) (2023)
- [Neural reproducing kernel Banach spaces and representer theorems for deep networks](https://mlsanity.com/paper/2403.08750/) (2024)
- [Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks](https://mlsanity.com/paper/2404.18769/) (2024)
- [Stability of Image-Reconstruction Algorithms](https://mlsanity.com/paper/2206.07128/) (2022)
- [Random Fourier Features for Asymmetric Kernels](https://mlsanity.com/paper/2209.08461/) (2022)
- [Dataset Condensation with Latent Quantile Matching](https://mlsanity.com/paper/2406.09860/) (2024)
- [Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space](https://mlsanity.com/paper/2307.01177/) (2023)
- [Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities](https://mlsanity.com/paper/2310.03696/) (2023)
- [ReLUs Are Sufficient for Learning Implicit Neural Representations](https://mlsanity.com/paper/2406.02529/) (2024)
- [Weighted Sobolev Approximation Rates for Neural Networks on Unbounded Domains](https://mlsanity.com/paper/2411.04108/) (2024)
- [From Kernel Methods to Neural Networks: A Unifying Variational Formulation](https://mlsanity.com/paper/2206.14625/) (2022)
- [Embeddings between Barron spaces with higher order activation functions](https://mlsanity.com/paper/2305.15839/) (2023)
- [Tractability of approximation by general shallow networks](https://mlsanity.com/paper/2308.03230/) (2023)
- [Hypothesis Spaces for Deep Learning](https://mlsanity.com/paper/2403.03353/) (2024)
- [Tighter Sparse Approximation Bounds for ReLU Neural Networks](https://mlsanity.com/paper/2110.03673/) (2021)
- [Random ReLU Neural Networks as Non-Gaussian Processes](https://mlsanity.com/paper/2405.10229/) (2024)
- [Solving Nonlinear PDEs with Sparse Radial Basis Function Networks](https://mlsanity.com/paper/2505.07765/) (2025)
- [The Mori-Zwanzig formulation of deep learning](https://mlsanity.com/paper/2209.05544/) (2022)
- [Nonuniform random feature models using derivative information](https://mlsanity.com/paper/2410.02132/) (2024)
- [A Lipschitz spaces view of infinitely wide shallow neural networks](https://mlsanity.com/paper/2410.14591/) (2024)
- [Mirror Descent on Reproducing Kernel Banach Spaces](https://mlsanity.com/paper/2411.11242/) (2024)
- [A Gap Between the Gaussian RKHS and Neural Networks: An Infinite-Center Asymptotic Analysis](https://mlsanity.com/paper/2502.16331/) (2025)
- [Manipulating Transformer-Based Models: Controllability, Steerability, and Robust Interventions](https://mlsanity.com/paper/2509.04549/) (2025)

## Abstract (arXiv)

Characterizing the function spaces corresponding to neural networks can provide a way to understand their properties. In this paper we discuss how the theory of reproducing kernel Banach spaces can be used to tackle this challenge. In particular, we prove a representer theorem for a wide class of reproducing kernel Banach spaces that admit a suitable integral representation and include one hidden layer neural networks of possibly infinite width. Further, we show that, for a suitable class of ReLU activation functions, the norm in the corresponding reproducing kernel Banach space can be characterized in terms of the inverse Radon transform of a bounded real measure, with norm given by the total variation norm of the measure. Our analysis simplifies and extends recent results in [34,29,30].