mlsanityData to 7 Oct 2026

The Promises and Pitfalls of Deep Kernel Learning

arXiv 2102.12108 ↗First posted 24 Feb 2021stat.ML cs.LGcited by 81

Not on the map. Why?

Abstract (arXiv)

Deep kernel learning (DKL) and related techniques aim to combine the representational power of neural networks with the reliable uncertainty estimates of Gaussian processes. One crucial aspect of these models is an expectation that, because they are treated as Gaussian process models optimized using the marginal likelihood, they are protected from overfitting. However, we identify situations where this is not the case. We explore this behavior, explain its origins and consider how it applies to real datasets. Through careful experimentation on the UCI, CIFAR-10, and the UTKFace datasets, we find that the overfitting from overparameterized maximum marginal likelihood, in which the model is "somewhat Bayesian", can in certain scenarios be worse than that from not being Bayesian at all. We explain how and when DKL can still be successful by investigating optimization dynamics. We also find that failures of DKL can be rectified by a fully Bayesian treatment, which leads to the desired performance improvements over standard neural networks and Gaussian processes.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 24 Feb 2021.

Builds on

34 of 56 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2015Deep Kernel Learningcited by 440
  9. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  10. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  11. 2014Scalable Variational Gaussian Process Classificationcited by 226
  12. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
22 more
  1. 2015Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)cited by 175
  2. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  3. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  4. 2019Practical Deep Learning with Bayesian Principlescited by 158
  5. 2017Doubly Stochastic Variational Inference for Deep Gaussian Processescited by 143
  6. 2016GPflow: A Gaussian process library using TensorFlowcited by 139
  7. 2016Stochastic Variational Deep Kernel Learningcited by 92
  8. 2014Manifold Gaussian Processes for Regressioncited by 79
  9. 2020Convergence of Sparse Variational Inference in Gaussian Processes Regressioncited by 53
  10. 2018Learning Invariances using the Marginal Likelihoodcited by 51
  11. 2017Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networkscited by 48
  12. 2015MCMC for Variationally Sparse Gaussian Processescited by 44
  13. 2017Convolutional Gaussian Processescited by 44
  14. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  15. 2020Uncertainty Estimation Using a Single Deep Deterministic Neural Networkcited by 37
  16. 2019Benchmarking the Neural Linear Model for Regressioncited by 31
  17. 2019Approximate Inference for Fully Bayesian Gaussian Process Regressioncited by 31
  18. 2015Thoughts on Massively Scalable Gaussian Processescited by 26
  19. 2019Bayesian Image Classification with Deep Convolutional Gaussian Processescited by 21
  20. 2019Bayesian Inference for Large Scale Image Classificationcited by 18
  21. 2018Calibrating Deep Convolutional Gaussian Processescited by 13
  22. 2016Fast methods for training Gaussian processes on large data setscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 16; 2023: 21; 2024: 16; 2025: 12; 2026: 8.

81 papers.

Sort:

By area:

  1. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  2. 2024Vanilla Bayesian Optimization Performs Great in High Dimensionscited by 87
  3. 2021Personalized Federated Learning with Gaussian Processescited by 76
  4. 2024Variational Bayesian Last Layerscited by 67
  5. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  6. 2024How to Build the Virtual Cell with Artificial Intelligence: Priorities and Opportunitiescited by 50
  7. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  8. 2023Deep networks for system identification: a Surveycited by 35
  9. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  10. 2022Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Predictioncited by 21
  11. 2023Learning a Depth Covariance Functioncited by 16
  12. 2022Environmental Sensor Placement with Convolutional Gaussian Neural Processescited by 14
38 more
  1. 2021Kernel Identification Through Transformerscited by 13
  2. 2025Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spacescited by 13
  3. 2023Bivariate Causal Discovery using Bayesian Model Selectioncited by 11
  4. 2023Uncertainty Estimation for Molecules: Desiderata and Methodscited by 11
  5. 2023RTDK-BO: High Dimensional Bayesian Optimization with Reinforced Transformer Deep kernelscited by 11
  6. 2021Model Selection for Bayesian Autoencoderscited by 10
  7. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  8. 2025Large language models as uncertainty-calibrated optimizers for experimental discoverycited by 10
  9. 2022Structural Kernel Search via Bayesian Optimization and Symbolical Optimal Transportcited by 8
  10. 2023A Unifying Perspective on Non-Stationary Kernels for Deeper Gaussian Processescited by 8
  11. 2021Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processescited by 7
  12. 2023Deep Learning with Kernels through RKHM and the Perron-Frobenius Operatorcited by 7
  13. 2024A prediction rigidity formalism for low-cost uncertainties in trained neural networkscited by 7
  14. 2024DKL-KAN: Scalable Deep Kernel Learning using Kolmogorov-Arnold Networkscited by 7
  15. 2022Deep Kernel Learning of Dynamical Models from High-Dimensional Noisy Datacited by 6
  16. 2023Guided Deep Kernel Learningcited by 6
  17. 2024Bayesian Exploration of Pre-trained Models for Low-shot Image Classificationcited by 6
  18. 2024Error Bounds For Gaussian Process Regression Under Bounded Support Noise With Applications To Safety Certificationcited by 6
  19. 2022Active learning in open experimental environments: selecting the right information channel(s) based on predictability in deep kernel learningcited by 5
  20. 2022Federated Bayesian Neural Regression: A Scalable Global Federated Gaussian Processcited by 5
  21. 2024Personalized Federated Learning of Probabilistic Models: A PAC-Bayesian Approachcited by 5
  22. 2022Sparse Gaussian Process Hyperparameters: Optimize or Integrate?cited by 4
  23. 2023Density Uncertainty Layers for Reliable Uncertainty Estimationcited by 4
  24. 2023Promises of Deep Kernel Learning for Control Synthesiscited by 4
  25. 2024Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processescited by 4
  26. 2022Learning Representation for Bayesian Optimization with Collision-free Regularizationcited by 3
  27. 2023Convolutional Deep Kernel Machinescited by 3
  28. 2023Thin and Deep Gaussian Processescited by 3
  29. 2024Active Learning with Fully Bayesian Neural Networks for Discontinuous and Nonstationary Datacited by 3
  30. 2024Towards Gaussian Process for operator learning: an uncertainty aware resolution independent operator learning algorithm for computational mechanicscited by 3
  31. 2024Learning-Based Shielding for Safe Autonomy under Unknown Dynamicscited by 3
  32. 2026Spectral methods: crucial for machine learning, natural for quantum computers?cited by 3
  33. 2022B'ezier Gaussian Processes for Tall and Wide Datacited by 2
  34. 2023Surrogate uncertainty estimation for your time series forecasting black-box: learn when to trustcited by 2
  35. 2023Evaluation of machine learning architectures on the quantification of epistemic and aleatoric uncertainties in complex dynamical systemscited by 2
  36. 2023Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Multiple Heterogeneous Datasetscited by 2
  37. 2024Recommendations for Baselines and Benchmarking Approximate Gaussian Processescited by 2
  38. 2025From Deep Additive Kernel Learning to Last-Layer Bayesian Neural Networks via Induced Prior Approximationcited by 2

50 most cited of 81 All 81 (JSON)