# A Gaussian Process View on Observation Noise and Initialization in Wide Neural Networks

arXiv: https://arxiv.org/abs/2502.01556 · HTML: https://mlsanity.com/paper/2502.01556/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Builds on (most cited first, 18 of 18)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [Tensor Programs II: Neural Tangent Kernel for Any Architecture](https://mlsanity.com/paper/2006.14548/) (2020)
- [On the linearity of large non-linear models: when and why the tangent kernel is constant](https://mlsanity.com/paper/2010.01092/) (2020)
- [Bayesian Deep Ensembles via the Neural Tangent Kernel](https://mlsanity.com/paper/2007.05864/) (2020)
- [On the training dynamics of deep networks with L2 regularization](https://mlsanity.com/paper/2006.08643/) (2020)
- [Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime](https://mlsanity.com/paper/2006.12297/) (2020)
- [Function-Space Regularization in Neural Networks: A Probabilistic Perspective](https://mlsanity.com/paper/2312.17162/) (2023)
- [Neural Latent Geometry Search: Product Manifold Inference via Gromov-Hausdorff-Informed Bayesian Optimization](https://mlsanity.com/paper/2309.04810/) (2023)
- [Asynchronous Distributed Variational Gaussian Processes for Regression](https://mlsanity.com/paper/1704.06735/) (2017)
- [Uncertainty Modeling in Graph Neural Networks via Stochastic Differential Equations](https://mlsanity.com/paper/2408.16115/) (2024)
- [Partially Stochastic Infinitely Deep Bayesian Neural Networks](https://mlsanity.com/paper/2402.03495/) (2024)
- [Epistemic Uncertainty and Observation Noise with the Neural Tangent Kernel](https://mlsanity.com/paper/2409.03953/) (2024)

## Built on by (most cited first, 5 of 5)

- [On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning](https://mlsanity.com/paper/2502.10818/) (2025)
- [Richer Bayesian Last Layers with Subsampled NTK Features](https://mlsanity.com/paper/2602.01279/) (2026)
- [Thompson Sampling in Function Spaces via Neural Operators](https://mlsanity.com/paper/2506.21894/) (2025)
- [The Neural Tangent Kernel for Classification](https://mlsanity.com/paper/2605.17606/) (2026)
- [Canonical Regularisation of Wide Feature-Learning Neural Networks](https://mlsanity.com/paper/2605.18180/) (2026)

## Abstract (arXiv)

Performing gradient descent in a wide neural network is equivalent to computing the posterior mean of a Gaussian Process with the Neural Tangent Kernel (NTK-GP), for a specific prior mean and with zero observation noise. However, existing formulations have two limitations: (i) the NTK-GP assumes noiseless targets, leading to misspecification on noisy data; (ii) the equivalence does not extend to arbitrary prior means, which are essential for well-specified models. To address (i), we introduce a regularizer into the training objective, showing its correspondence to incorporating observation noise in the NTK-GP. To address (ii), we propose a textitshifted network that enables arbitrary prior means and allows obtaining the posterior mean with gradient descent on a single network, without ensembling or kernel inversion. We validate our results with experiments across datasets and architectures, showing that this approach removes key obstacles to the practical use of NTK-GP equivalence in applied Gaussian process modeling.