Analysis of Structured Deep Kernel Networks
Not on the map. Why?
Abstract (arXiv)
In this paper, we leverage a recent deep kernel representer theorem to connect kernel based learning and (deep) neural networks in order to understand their interplay. In particular, we show that the use of special types of kernels yields models reminiscent of neural networks that are founded in the same theoretical framework of classical kernel methods, while benefiting from the computational advantages of deep neural networks. Especially the introduced Structured Deep Kernel Networks (SDKNs) can be viewed as neural networks (NNs) with optimizable activation functions obeying a representer theorem. This link allows us to analyze also NNs within the framework of kernel networks. We prove analytic properties of the SDKNs which show their universal approximation properties in three different asymptotic regimes of unbounded number of centers, width and depth. Especially in the case of unbounded depth, more accurate constructions can be achieved using fewer layers compared to corresponding constructions for ReLU neural networks. This is made possible by leveraging properties of kernel approximation.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 62 references that are in the map. First posted on arXiv 15 May 2021.
Builds on
24 of 62 references.
References · 24
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2016Error bounds for approximations with deep ReLU networkscited by 567
- 2015Deep Kernel Learningcited by 440
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2016Benefits of depth in neural networkscited by 248
- 2019Universal Approximation with Deep Narrow Networkscited by 198
- 2015The Power of Depth for Feedforward Neural Networkscited by 181
- 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
- 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
- 2019Deep Neural Network Approximation Theorycited by 92
- 2020Kernel methods through the roof: handling billions of points efficientlycited by 88
12 more
- 2018Kernel Flows: from learning kernels from data into the abysscited by 49
- 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
- 2018A representer theorem for deep neural networkscited by 38
- 2017Diving into the shallows: a computational perspective on large-scale shallow learningcited by 30
- 2019Smooth function approximation by deep neural networks with general activation functionscited by 29
- 2017A representer theorem for deep kernel learningcited by 26
- 2020Learning dynamical systems from data: a simple cross-validation perspectivecited by 23
- 2020Approximating smooth functions by deep neural networks with sigmoid activation functioncited by 18
- 2020Do ideas have shape? Idea registration as the continuous limit of artificial neural networkscited by 16
- 2021Data-driven geophysical forecasting: Simple, low-cost, and accurate baselines with kernel methodscited by 15
- 2016Learning with Hierarchical Gaussian Kernelscited by 4
- 2021Structured Deep Kernel Networks for Data-Driven Closure Terms of Turbulent Flowscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.