Data to 7 Oct 2026

Gaussian Process Behaviour in Wide Deep Neural Networks

arXiv 1804.11271 ↗First posted 30 Apr 2018stat.ML cs.LGcited by 314

Not on the map. Why?

Abstract (arXiv)

Whilst deep neural networks have shown great empirical success, there is still much work to be done to understand their theoretical properties. In this paper, we study the relationship between random, wide, fully connected, feedforward networks with more than one hidden layer and Gaussian processes with a recursive kernel definition. We show that, under broad conditions, as we make the architecture increasingly wide, the implied random function converges in distribution to a Gaussian process, formalising and extending existing results by Neal (1996) to deep networks. To evaluate convergence rates empirically, we use maximum mean discrepancy. We then compare finite Bayesian deep networks from the literature to Gaussian processes in terms of the key predictive quantities of interest, finding that in some cases the agreement can be very close. We discuss the desirability of Gaussian process behaviour and review non-Gaussian alternative models from the literature.

Key facts. Cited by 314 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 72; 2022: 70; 2023: 60; 2024: 40; 2025: 41; 2026: 30.

314 papers.

Sort:

By area:

2021Explaining Neural Scaling Lawscited by 353
2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
2021Bayesian Neural Network Priors Revisitedcited by 121
2021Epistemic Neural Networkscited by 120
2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
2021Priors in Bayesian Deep Learning: A Reviewcited by 101
2022A Solvable Model of Neural Scaling Lawscited by 101
2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74

38 more

2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
2022Fast Finite Width Neural Tangent Kernelcited by 62
2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
2025Deep Learning is Not So Mysterious or Differentcited by 55
2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 47
2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
2024Neural Redshift: Random Networks are not Random Functionscited by 43
2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
2023Simplifying Transformer Blockscited by 37
2023Deep networks for system identification: a Surveycited by 35
2023Deterministic equivalent and error universality of deep random features learningcited by 35
2021Cross-modal Adversarial Reprogrammingcited by 34
2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
2021Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlatorscited by 28
2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 28
2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
2023Deep neural networks have an inbuilt Occam's razorcited by 28
2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
2022NeuralEF: Deconstructing Kernels by Deep Neural Networkscited by 26
2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
2022Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernelscited by 25
2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
2021Nonperturbative renormalization for the neural network-QFT correspondencecited by 24
2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 24

50 most cited of 314 All 314 (JSON)