# A representer theorem for deep neural networks

arXiv: https://arxiv.org/abs/1802.09210 · HTML: https://mlsanity.com/paper/1802.09210/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2018.

## Built on by (most cited first, 30 of 38)

- [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021)
- [A Neural-Network-Based Convex Regularizer for Inverse Problems](https://mlsanity.com/paper/2211.12461/) (2022)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions](https://mlsanity.com/paper/2210.16222/) (2022)
- [MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining](https://mlsanity.com/paper/2110.08009/) (2021)
- [On the Number of Regions of Piecewise Linear Neural Networks](https://mlsanity.com/paper/2206.08615/) (2022)
- [Sampling weights of deep neural networks](https://mlsanity.com/paper/2306.16830/) (2023)
- [Approximation of Lipschitz Functions using Deep Spline Neural Networks](https://mlsanity.com/paper/2204.06233/) (2022)
- [Duality for Neural Networks through Reproducing Kernel Banach Spaces](https://mlsanity.com/paper/2211.05020/) (2022)
- [A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Moments](https://mlsanity.com/paper/2202.08325/) (2022)
- [Fully-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on Helgason-Fourier Analysis](https://mlsanity.com/paper/2203.01631/) (2022)
- [Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groups](https://mlsanity.com/paper/2205.14819/) (2022)
- [Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks](https://mlsanity.com/paper/2404.18769/) (2024)
- [Stability of Image-Reconstruction Algorithms](https://mlsanity.com/paper/2206.07128/) (2022)
- [CAS4DL: Christoffel Adaptive Sampling for function approximation via Deep Learning](https://mlsanity.com/paper/2208.12190/) (2022)
- [Max-Affine Spline Insights Into Deep Network Pruning](https://mlsanity.com/paper/2101.02338/) (2021)
- [Sparse Deep Neural Network for Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2207.13266/) (2022)
- [Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptrons](https://mlsanity.com/paper/2409.10463/) (2024)
- [Fast Jacobian-Vector Product for Deep Networks](https://mlsanity.com/paper/2104.00219/) (2021)
- [Measuring Complexity of Learning Schemes Using Hessian-Schatten Total Variation](https://mlsanity.com/paper/2112.06209/) (2021)
- [Optimal Activation Functions for the Random Features Regression Model](https://mlsanity.com/paper/2206.01332/) (2022)
- [Hypothesis Spaces for Deep Learning](https://mlsanity.com/paper/2403.03353/) (2024)
- [Analysis of Structured Deep Kernel Networks](https://mlsanity.com/paper/2105.07228/) (2021)
- [Sparsest Univariate Learning Models Under Lipschitz Constraint](https://mlsanity.com/paper/2112.13542/) (2021)
- [A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks](https://mlsanity.com/paper/2402.15984/) (2024)
- [Parseval Convolution Operators and Neural Networks](https://mlsanity.com/paper/2408.09981/) (2024)
- [Representation and Regression Problems in Neural Networks: Relaxation, Generalization, and Numerics](https://mlsanity.com/paper/2412.01619/) (2024)
- [Representer Point Selection for Explaining Regularized High-dimensional Models](https://mlsanity.com/paper/2305.20002/) (2023)
- [Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures](https://mlsanity.com/paper/2408.13114/) (2024)
- [Training Neural Networks for Solving 1-D Optimal Piecewise Linear Approximation](https://mlsanity.com/paper/2110.08259/) (2021)

## Abstract (arXiv)

We propose to optimize the activation functions of a deep neural network by adding a corresponding functional regularization to the cost function. We justify the use of a second-order total-variation criterion. This allows us to derive a general representer theorem for deep neural networks that makes a direct connection with splines and sparsity. Specifically, we show that the optimal network configuration can be achieved with activation functions that are nonuniform linear splines with adaptive knots. The bottom line is that the action of each neuron is encoded by a spline whose parameters (including the number of knots) are optimized during the training procedure. The scheme results in a computational structure that is compatible with the existing deep-ReLU, parametric ReLU, APL (adaptive piecewise-linear) and MaxOut architectures. It also suggests novel optimization challenges, while making the link with ℓ1 minimization and sparsity-promoting techniques explicit.