# A unifying representer theorem for inverse problems and machine learning

arXiv: https://arxiv.org/abs/1903.00687 · HTML: https://mlsanity.com/paper/1903.00687/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2019.

## Built on by (most cited first, 24 of 24)

- [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021)
- [Understanding neural networks with reproducing kernel Banach spaces](https://mlsanity.com/paper/2109.09710/) (2021)
- [Two-layer neural networks with values in a Banach space](https://mlsanity.com/paper/2105.02095/) (2021)
- [Ridges, Neural Networks, and the Radon Transform](https://mlsanity.com/paper/2203.02543/) (2022)
- [Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compression](https://mlsanity.com/paper/2305.16534/) (2023)
- [Sparse Representer Theorems for Learning in Reproducing Kernel Banach Spaces](https://mlsanity.com/paper/2305.12584/) (2023)
- [Neural reproducing kernel Banach spaces and representer theorems for deep networks](https://mlsanity.com/paper/2403.08750/) (2024)
- [Stability of Image-Reconstruction Algorithms](https://mlsanity.com/paper/2206.07128/) (2022)
- [Weighted variation spaces and approximation by shallow ReLU networks](https://mlsanity.com/paper/2307.15772/) (2023)
- [Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities](https://mlsanity.com/paper/2310.03696/) (2023)
- [From Kernel Methods to Neural Networks: A Unifying Variational Formulation](https://mlsanity.com/paper/2206.14625/) (2022)
- [Sparsest Univariate Learning Models Under Lipschitz Constraint](https://mlsanity.com/paper/2112.13542/) (2021)
- [Efficient Algorithms for Regularized Nonnegative Scale-invariant Low-rank Approximation Models](https://mlsanity.com/paper/2403.18517/) (2024)
- [Gradient Descent in Neural Networks as Sequential Learning in RKBS](https://mlsanity.com/paper/2302.00205/) (2023)
- [Analysis of Regularized Learning in Banach Spaces for Linear-functional Data](https://mlsanity.com/paper/2109.03159/) (2021)
- [Optimal Learning](https://mlsanity.com/paper/2203.15994/) (2022)
- [Delaunay-Triangulation-Based Learning with Hessian Total-Variation Regularization](https://mlsanity.com/paper/2208.07787/) (2022)
- [Computerized Tomography and Reproducing Kernels](https://mlsanity.com/paper/2311.07465/) (2023)
- [Novel Kernel Models and Exact Representor Theory for Neural Networks Beyond the Over-Parameterized Regime](https://mlsanity.com/paper/2405.15254/) (2024)
- [Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Bias](https://mlsanity.com/paper/2406.09194/) (2024)
- [Global-to-Local Support Spectrums for Language Model Explainability](https://mlsanity.com/paper/2408.05976/) (2024)
- [Decomposition of one-layer neural networks via the infinite sum of reproducing kernel Banach spaces](https://mlsanity.com/paper/2409.18132/) (2024)
- [The Effects of Multi-Task Learning on ReLU Neural Network Functions](https://mlsanity.com/paper/2410.21696/) (2024)
- [Generalized Splines and Gaussian Processes](https://mlsanity.com/paper/2608.28446/) (2026)

## Abstract (arXiv)

The standard approach for dealing with the ill-posedness of the training problem in machine learning and/or the reconstruction of a signal from a limited number of measurements is regularization. The method is applicable whenever the problem is formulated as an optimization task. The standard strategy consists in augmenting the original cost functional by an energy that penalizes solutions with undesirable behavior. The effect of regularization is very well understood when the penalty involves a Hilbertian norm. Another popular configuration is the use of an ℓ1-norm (or some variant thereof) that favors sparse solutions. In this paper, we propose a higher-level formulation of regularization within the context of Banach spaces. We present a general representer theorem that characterizes the solutions of a remarkably broad class of optimization problems. We then use our theorem to retrieve a number of known results in the literature---e.g., the celebrated representer theorem of machine leaning for RKHS, Tikhonov regularization, representer theorems for sparsity promoting functionals, the recovery of spikes---as well as a few new ones.