mlsanityData to 7 Oct 2026

Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes

arXiv 1910.12478 ↗First posted 28 Oct 2019cs.NE cond-mat.dis-nn cs.LGcited by 148

Not on the map. Why?

Abstract (arXiv)

Wide neural networks with random weights and biases are Gaussian processes, as originally observed by Neal (1995) and more recently by Lee et al. (2018) and Matthews et al. (2018) for deep fully-connected networks, as well as by Novak et al. (2019) and Garriga-Alonso et al. (2019) for deep convolutional networks. We show that this Neural Network-Gaussian Process correspondence surprisingly extends to all modern feedforward or recurrent neural networks composed of multilayer perceptron, RNNs (e.g. LSTMs, GRUs), (nD or graph) convolution, pooling, skip connection, attention, batch normalization, and/or layer normalization. More generally, we introduce a language for expressing neural network computations, and our result encompasses all such expressible neural networks. This work serves as a tutorial on the *tensor programs* technique formulated in Yang (2019) and elucidates the Gaussian Process results obtained there. We provide open-source implementations of the Gaussian Process kernels of simple RNN, GRU, transformer, and batchnorm+ReLU network at github.com/thegregyang/GP4A.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 29; 2022: 26; 2023: 29; 2024: 24; 2025: 19; 2026: 20.

148 papers.

Sort:

By area:

  1. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  2. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  3. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  4. 2021Bayesian Neural Network Priors Revisitedcited by 121
  5. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  6. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  7. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  8. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  9. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  10. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  11. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  12. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
38 more
  1. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  2. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  3. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  4. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  5. 2023Simplifying Transformer Blockscited by 37
  6. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  7. 2023Deep networks for system identification: a Surveycited by 35
  8. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  9. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  10. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  11. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  12. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  13. 2021Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlatorscited by 28
  14. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  15. 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
  16. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  17. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  18. 2021Nonperturbative renormalization for the neural network-QFT correspondencecited by 24
  19. 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 24
  20. 2022Neural Tangent Kernel: A Surveycited by 22
  21. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  22. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  23. 2023Neural Network Field Theories: Non-Gaussianity, Actions, and Localitycited by 21
  24. 2023Leave-one-out Distinguishability in Machine Learningcited by 20
  25. 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 18
  26. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  27. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 18
  28. 2023Differentially Private Neural Tangent Kernels for Privacy-Preserving Data Generationcited by 15
  29. 2024(Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Modelscited by 15
  30. 2021Large-width functional asymptotics for deep Gaussian neural networkscited by 14
  31. 2021Learning curves for Gaussian process regression with power-law priors and targetscited by 14
  32. 2023Prediction Error-based Classification for Class-Incremental Learningcited by 14
  33. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  34. 2021Why flatness does and does not correlate with generalization for deep neural networkscited by 13
  35. 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
  36. 2023Structures of Neural Network Effective Theoriescited by 13
  37. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  38. 2021Reverse Engineering the Neural Tangent Kernelcited by 12

50 most cited of 148 All 148 (JSON)