Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machines
Not on the map. Why?
Abstract (arXiv)
Recent work developed convolutional deep kernel machines, achieving 92.7% test accuracy on CIFAR-10 using a ResNet-inspired architecture, which is SOTA for kernel methods. However, this still lags behind neural networks, which easily achieve over 94% test accuracy with similar architectures. In this work we introduce several modifications to improve the convolutional deep kernel machine's generalisation, including stochastic kernel regularisation, which adds noise to the learned Gram matrices during training. The resulting model achieves 94.5% test accuracy on CIFAR-10. This finding has important theoretical and practical implications, as it demonstrates that the ability to perform well on complex tasks like image classification is not unique to neural networks. Instead, other approaches including deep kernel methods can achieve excellent performance on such tasks, as long as they have the capacity to learn representations from data.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 8 Oct 2024.
Builds on
36 of 55 references.
References · 36
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2015Deep Kernel Learningcited by 440
- 2021The Principles of Deep Learning Theorycited by 169
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
- 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
- 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
- 2017Doubly Stochastic Variational Inference for Deep Gaussian Processescited by 143
- 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
24 more
- 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
- 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
- 2020Kernel Operations on the GPU, with Autodiff, without Memory Overflowscited by 101
- 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
- 2019Non-Gaussian processes and neural networks at finite widthscited by 78
- 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
- 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
- 2020Neural Networks and Quantum Field Theorycited by 58
- 2020Neural Kernels Without Tangentscited by 56
- 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
- 2014Convolutional Kernel Networkscited by 47
- 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
- 2020Deep kernel processescited by 34
- 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
- 2016End-to-End Kernel Learning with Supervised Convolutional Kernel Networkscited by 28
- 2019Finite size corrections for neural network Gaussian processescited by 22
- 2019Bayesian Image Classification with Deep Convolutional Gaussian Processescited by 21
- 2021A theory of representation learning gives a deep generalisation of kernel methodscited by 15
- 2022Low-Precision Arithmetic for Fast Gaussian Processescited by 10
- 2023Kernel Regression with Infinite-Width Neural Networks on Millions of Examplescited by 9
- 2021A variational approximate posterior for the deep Wishart processcited by 6
- 2023Guided Deep Kernel Learningcited by 6
- 2017Optimizing Kernel Machines using Deep Learningcited by 3
- 2023An Improved Variational Approximate Posterior for the Deep Wishart Processcited by 3
Built on by
Not cited yet.