mlsanityData to 7 Oct 2026

A representer theorem for deep neural networks

arXiv 1802.09210 ↗First posted 26 Feb 2018stat.ML cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

We propose to optimize the activation functions of a deep neural network by adding a corresponding functional regularization to the cost function. We justify the use of a second-order total-variation criterion. This allows us to derive a general representer theorem for deep neural networks that makes a direct connection with splines and sparsity. Specifically, we show that the optimal network configuration can be achieved with activation functions that are nonuniform linear splines with adaptive knots. The bottom line is that the action of each neuron is encoded by a spline whose parameters (including the number of knots) are optimized during the training procedure. The scheme results in a computational structure that is compatible with the existing deep-ReLU, parametric ReLU, APL (adaptive piecewise-linear) and MaxOut architectures. It also suggests novel optimization challenges, while making the link with ℓ1 minimization and sparsity-promoting techniques explicit.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 13; 2023: 3; 2024: 11; 2025: 0; 2026: 3.

38 papers.

Sort:

  1. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  2. 2022A Neural-Network-Based Convex Regularizer for Inverse Problemscited by 40
  3. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  4. 2021MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retrainingcited by 23
  5. 2022Improving Lipschitz-Constrained Neural Networks by Learning Activation Functionscited by 23
  6. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  7. 2023Sampling weights of deep neural networkscited by 20
  8. 2022Approximation of Lipschitz Functions using Deep Spline Neural Networkscited by 15
  9. 2022Duality for Neural Networks through Reproducing Kernel Banach Spacescited by 15
  10. 2022A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Momentscited by 14
  11. 2022Fully-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on Helgason-Fourier Analysiscited by 14
  12. 2022Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groupscited by 11
26 more
  1. 2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
  2. 2022Stability of Image-Reconstruction Algorithmscited by 7
  3. 2022CAS4DL: Christoffel Adaptive Sampling for function approximation via Deep Learningcited by 6
  4. 2021Max-Affine Spline Insights Into Deep Network Pruningcited by 5
  5. 2022Sparse Deep Neural Network for Nonlinear Partial Differential Equationscited by 5
  6. 2024Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptronscited by 5
  7. 2021Fast Jacobian-Vector Product for Deep Networkscited by 4
  8. 2021Measuring Complexity of Learning Schemes Using Hessian-Schatten Total Variationcited by 4
  9. 2022Optimal Activation Functions for the Random Features Regression Modelcited by 4
  10. 2024Hypothesis Spaces for Deep Learningcited by 4
  11. 2021Analysis of Structured Deep Kernel Networkscited by 2
  12. 2021Sparsest Univariate Learning Models Under Lipschitz Constraintcited by 2
  13. 2024A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networkscited by 2
  14. 2024Parseval Convolution Operators and Neural Networkscited by 2
  15. 2024Representation and Regression Problems in Neural Networks: Relaxation, Generalization, and Numericscited by 2
  16. 2023Representer Point Selection for Explaining Regularized High-dimensional Modelscited by 1
  17. 2024Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architecturescited by 1
  18. 2021Training Neural Networks for Solving 1-D Optimal Piecewise Linear Approximationcited by 0
  19. 2022Real Image Super-Resolution using GAN through modeling of LR and HR processcited by 0
  20. 2024Novel Kernel Models and Exact Representor Theory for Neural Networks Beyond the Over-Parameterized Regimecited by 0
  21. 2024On the Geometry of Deep Learningcited by 0
  22. 2024Global-to-Local Support Spectrums for Language Model Explainabilitycited by 0
  23. 2024A resource-efficient model for deep kernel learningcited by 0
  24. 2026Penalizing Localized Dirichlet Energies in Low Rank Tensor Productscited by 0
  25. 2026Kernel-Based Learning of Chest X-ray Images for Predicting ICU Escalation among COVID-19 Patientscited by 0
  26. 2026A Function-Space Approach to the Statistical Mechanics of Learning Dynamicscited by 0