mlsanityData to 14 Oct 2026

Deep Neural Networks as Gaussian Processes

arXiv 1711.00165 ↗First posted 1 Nov 2017stat.ML cs.LGcited by 622

Not on the map. Why?

Abstract (arXiv)

It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.

Key facts. Cited by 622 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 1; 2021: 125; 2022: 113; 2023: 113; 2024: 99; 2025: 102; 2026: 68.

622 papers.

Sort:

By area:

  1. 2021Explaining Neural Scaling Lawscited by 363
  2. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  3. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  4. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  5. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  6. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  7. 2021Bayesian Neural Network Priors Revisitedcited by 121
  8. 2022Diffusion Probabilistic Model Made Slimcited by 121
  9. 2021Epistemic Neural Networkscited by 120
  10. 2022A Solvable Model of Neural Scaling Lawscited by 116
  11. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  12. 2023A Spectral Condition for Feature Learningcited by 109
38 more
  1. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  2. 2021Repulsive Deep Ensembles are Bayesiancited by 104
  3. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  4. 2018EnsembleDAgger: A Bayesian Approach to Safe Imitation Learningcited by 100
  5. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
  6. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
  7. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  8. 2024Model Collapse Demystified: The Case of Regressioncited by 82
  9. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
  10. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 70
  11. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
  12. 2024Scalable Optimization in the Modular Normcited by 68
  13. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  14. 2024Strong Model Collapsecited by 65
  15. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  16. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  17. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  18. 2025Deep Learning is Not So Mysterious or Differentcited by 56
  19. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  20. 2021Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptationcited by 50
  21. 2021Understanding neural networks with reproducing kernel Banach spacescited by 50
  22. 2023A Study of Bayesian Neural Network Surrogates for Bayesian Optimizationcited by 50
  23. 2021Faster Single-loop Algorithms for Minimax Optimization without Strong Concavitycited by 49
  24. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 49
  25. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  26. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
  27. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  28. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 43
  29. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  30. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  31. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  32. 2021Learning effective stochastic differential equations from microscopic simulations: linking stochastic numerics to deep learningcited by 39
  33. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  34. 2021Learning Functional Priors and Posteriors from Data and Physicscited by 37
  35. 2023Deep networks for system identification: a Surveycited by 37
  36. 2023Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzlecited by 37
  37. 2023Simplifying Transformer Blockscited by 37
  38. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36

50 most cited of 622 All 622 (JSON)