mlsanityData to 7 Oct 2026

Guided Deep Kernel Learning

arXiv 2302.09574 ↗First posted 19 Feb 2023cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Combining Gaussian processes with the expressive power of deep neural networks is commonly done nowadays through deep kernel learning (DKL). Unfortunately, due to the kernel optimization process, this often results in losing their Bayesian benefits. In this study, we present a novel approach for learning deep kernels by utilizing infinite-width neural networks. We propose to use the Neural Network Gaussian Process (NNGP) model as a guide to the DKL model in the optimization process. Our approach harnesses the reliable uncertainty estimation of the NNGPs to adapt the DKL target confidence when it encounters novel data points. As a result, we get the best of both worlds, we leverage the Bayesian behavior of the NNGP, namely its robustness to overfitting, and accurate uncertainty estimation, while maintaining the generalization abilities, scalability, and flexibility of deep kernels. Empirically, we show on multiple benchmark datasets of varying sizes and dimensionality, that our method is robust to overfitting, has good predictive performance, and provides reliable uncertainty estimations.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 44 of its 66 references that are in the map. First posted on arXiv 19 Feb 2023.

Builds on

44 of 66 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2016Wide Residual Networkscited by 3,057
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2017Deep Neural Networks as Gaussian Processescited by 617
  7. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  8. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  9. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  10. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  11. 2015Deep Kernel Learningcited by 440
  12. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
32 more
  1. 2013Gaussian Processes for Big Datacited by 377
  2. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  3. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  4. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  5. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  6. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  7. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  8. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  9. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  10. 2017Doubly Stochastic Variational Inference for Deep Gaussian Processescited by 143
  11. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  12. 2019Functional Variational Bayesian Neural Networkscited by 127
  13. 2016Stochastic Variational Deep Kernel Learningcited by 92
  14. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  15. 2021On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertaintycited by 87
  16. 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
  17. 2014Manifold Gaussian Processes for Regressioncited by 79
  18. 2021Personalized Federated Learning with Gaussian Processescited by 76
  19. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  20. 2018Dirichlet-based Gaussian Processes for Large-scale Calibrated Classificationcited by 50
  21. 2017Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networkscited by 48
  22. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  23. 2019Function Space Particle Optimization for Bayesian Neural Networkscited by 41
  24. 2020Deep kernel processescited by 34
  25. 2020Bayesian Few-Shot Classification with One-vs-Each P'olya-Gamma Augmented Gaussian Processescited by 32
  26. 2021GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learningcited by 26
  27. 2015Dependent Multinomial Models Made Easy: Stick Breaking with the P'olya-Gamma Augmentationcited by 25
  28. 2020On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimationcited by 23
  29. 2019Deep Kernels with Probabilistic Embeddings for Small-Data Learningcited by 7
  30. 2021A variational approximate posterior for the deep Wishart processcited by 6
  31. 2020Deep Latent-Variable Kernel Learningcited by 5
  32. 2022Functional Ensemble Distillationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 2.

6 papers.

  1. 2024Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learningcited by 11
  2. 2024DKL-KAN: Scalable Deep Kernel Learning using Kolmogorov-Arnold Networkscited by 7
  3. 2023Convolutional Deep Kernel Machinescited by 3
  4. 2025Few-Shot Speech Deepfake Detection Adaptation with Gaussian Processescited by 1
  5. 2024Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machinescited by 0
  6. 2025Scalable Deep Basis Kernel Gaussian Processescited by 0