mlsanityData to 7 Oct 2026

Deep Neural Networks as Point Estimates for Deep Gaussian Processes

arXiv 2105.04504 ↗First posted 10 May 2021stat.ML cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Neural networks and Gaussian processes are complementary in their strengths and weaknesses. Having a better understanding of their relationship comes with the promise to make each method benefit from the strengths of the other. In this work, we establish an equivalence between the forward passes of neural networks and (deep) sparse Gaussian process models. The theory we develop is based on interpreting activation functions as interdomain inducing features through a rigorous analysis of the interplay between activation functions and kernels. This results in models that can either be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased prediction accuracy. These claims are supported by experimental results on regression and classification datasets.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 55 of its 89 references that are in the map. First posted on arXiv 10 May 2021.

Builds on

55 of 89 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2016Variational Inference: A Review for Statisticianscited by 1,466
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  10. 2017Deep Neural Networks as Gaussian Processescited by 617
  11. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  12. 2013Gaussian Processes for Big Datacited by 377
38 more
  1. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  2. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  3. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  4. 2014Scalable Variational Gaussian Process Classificationcited by 226
  5. 2013Predicting Parameters in Deep Learningcited by 223
  6. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 188
  7. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  8. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  9. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  10. 2017Doubly Stochastic Variational Inference for Deep Gaussian Processescited by 143
  11. 2016GPflow: A Gaussian process library using TensorFlowcited by 139
  12. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  13. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  14. 2013Gaussian Process Kernels for Pattern Discovery and Extrapolationcited by 131
  15. 2019Exact Gaussian Processes on a Million Data Pointscited by 118
  16. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  17. 2014Hilbert Space Methods for Reduced-Rank Gaussian Process Regressioncited by 88
  18. 2013Structure Discovery in Nonparametric Regression through Compositional Kernel Searchcited by 86
  19. 2016Variational Fourier features for Gaussian processescited by 85
  20. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  21. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  22. 2020Mat'ern Gaussian processes on Riemannian manifoldscited by 82
  23. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  24. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  25. 2019Rates of Convergence for Sparse Variational Gaussian Process Regressioncited by 62
  26. 2015On Sparse variational methods and the Kullback-Leibler divergence between stochastic processescited by 56
  27. 2018Learning Invariances using the Marginal Likelihoodcited by 51
  28. 2020A Framework for Interdomain and Multioutput Gaussian Processescited by 47
  29. 2017Convolutional Gaussian Processescited by 44
  30. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  31. 2020Sparse Gaussian Processes with Spherical Harmonic Featurescited by 43
  32. 2016Deep Gaussian Processes for Regression using Approximate Expectation Propagationcited by 39
  33. 2019On the marginal likelihood and cross-validationcited by 36
  34. 2014Avoiding pathologies in very deep networkscited by 33
  35. 2016Random Feature Expansions for Deep Gaussian Processescited by 32
  36. 2020A Tutorial on Sparse Gaussian Processes and Variational Inferencecited by 29
  37. 2018Variational Bayesian dropout: pitfalls and fixescited by 22
  38. 2019Bayesian Image Classification with Deep Convolutional Gaussian Processescited by 21

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 4; 2023: 11; 2024: 3; 2025: 2; 2026: 1.

26 papers.

Sort:

By area:

  1. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  2. 2022The Unreasonable Effectiveness of Deep Evidential Regressioncited by 40
  3. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  4. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  5. 2022Learning Invariant Weights in Neural Networkscited by 16
  6. 2023Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processescited by 6
  7. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  8. 2023Probabilistic Self-supervised Learning via Scoring Rules Minimizationcited by 4
  9. 2023On permutation symmetries in Bayesian neural network posteriors: a variational perspectivecited by 4
  10. 2024Gaussian Measures Conditioned on Nonlinear Observations: Consistency, MAP Estimators, and Simulationcited by 4
  11. 2023Convolutional Deep Kernel Machinescited by 3
  12. 2023Gaussian Process-Gated Hierarchical Mixtures of Expertscited by 2
14 more
  1. 2023Efficient Uncertainty Estimation with Gaussian Process for Reliable Dialog Response Retrievalcited by 2
  2. 2023Sparse Gaussian Processes with Spherical Harmonic Features Revisitedcited by 2
  3. 2025From Deep Additive Kernel Learning to Last-Layer Bayesian Neural Networks via Induced Prior Approximationcited by 2
  4. 2021Conditional Deep Gaussian Processes: empirical Bayes hyperdata learningcited by 1
  5. 2021GPEX, A Framework For Interpreting Artificial Neural Networkscited by 1
  6. 2022On Connecting Deep Trigonometric Networks with Deep Gaussian Processes: Covariance, Expressivity, and Neural Tangent Kernelcited by 1
  7. 2023Spherical Inducing Features for Orthogonally-Decoupled Gaussian Processescited by 1
  8. 2024Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?cited by 1
  9. 2025Bayesian neural networks with interpretable priors from Mercer kernelscited by 1
  10. 2022Sharp asymptotics on the compression of two-layer neural networkscited by 0
  11. 2023On Neural Networks as Infinite Tree-Structured Probabilistic Graphical Modelscited by 0
  12. 2023SemiGPC: Distribution-Aware Label Refinement for Imbalanced Semi-Supervised Learning Using Gaussian Processescited by 0
  13. 2024Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradientcited by 0
  14. 2026Surrogate-Gated Generation and Foundation-Model Embeddings for Bayesian Materials Designcited by 0