Convolutional Deep Kernel Machines
Not on the map. Why?
Abstract (arXiv)
Standard infinite-width limits of neural networks sacrifice the ability for intermediate layers to learn representations from data. Recent work (A theory of representation learning gives a deep generalisation of kernel methods, Yang et al. 2023) modified the Neural Network Gaussian Process (NNGP) limit of Bayesian neural networks so that representation learning is retained. Furthermore, they found that applying this modified limit to a deep Gaussian process gives a practical learning algorithm which they dubbed the deep kernel machine (DKM). However, they only considered the simplest possible setting: regression in small, fully connected networks with e.g. 10 input features. Here, we introduce convolutional deep kernel machines. This required us to develop a novel inter-domain inducing point approximation, as well as introducing and experimentally assessing a number of techniques not previously seen in DKMs, including analogues to batch normalisation, different likelihoods, and different types of top-layer. The resulting model trains in roughly 77 GPU hours, achieving around 99% test accuracy on MNIST, 72% on CIFAR-100, and 92.7% on CIFAR-10, which is SOTA for kernel methods.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 47 of its 65 references that are in the map. First posted on arXiv 18 Sep 2023.
Builds on
47 of 65 references.
References · 47
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2017Deep Neural Networks as Gaussian Processescited by 622
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
- 2015Deep Kernel Learningcited by 445
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
- 2020Feature Learning in Infinite-Width Neural Networkscited by 190
- 2021The Principles of Deep Learning Theorycited by 173
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
- 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
- 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
35 more
- 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
- 2016Stochastic Variational Deep Kernel Learningcited by 93
- 2016Variational Fourier features for Gaussian processescited by 85
- 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
- 2014Manifold Gaussian Processes for Regressioncited by 80
- 2019Non-Gaussian processes and neural networks at finite widthscited by 78
- 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
- 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
- 2020Neural Kernels Without Tangentscited by 56
- 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
- 2017Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networkscited by 48
- 2014Convolutional Kernel Networkscited by 47
- 2017Convolutional Gaussian Processescited by 44
- 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
- 2018Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Modelscited by 38
- 2020Deep kernel processescited by 34
- 2016End-to-End Kernel Learning with Supervised Convolutional Kernel Networkscited by 28
- 2022Benchopt: Reproducible, efficient and collaborative optimization benchmarkscited by 27
- 2017A representer theorem for deep kernel learningcited by 26
- 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
- 2019Finite size corrections for neural network Gaussian processescited by 22
- 2018Deep convolutional Gaussian processescited by 21
- 2019Bayesian Image Classification with Deep Convolutional Gaussian Processescited by 21
- 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
- 2020Wide Neural Networks with Bottlenecks are Deep Gaussian Processescited by 20
- 2021A theory of representation learning gives a deep generalisation of kernel methodscited by 15
- 2022Low-Precision Arithmetic for Fast Gaussian Processescited by 10
- 2020Inter-domain Deep Gaussian Processescited by 9
- 2023Kernel Regression with Infinite-Width Neural Networks on Millions of Examplescited by 9
- 2019Recurrent Kernel Networkscited by 6
- 2021A variational approximate posterior for the deep Wishart processcited by 6
- 2023Guided Deep Kernel Learningcited by 6
- 2023Deep Kernel Principal Component Analysis for Multi-level Feature Learningcited by 6
- 2023An Improved Variational Approximate Posterior for the Deep Wishart Processcited by 3
- 2022Deep Layer-wise Networks Have Closed-Form Weightscited by 2
Built on by
Citing papers per year: 2024: 3.
3 papers.