mlsanityData to 14 Oct 2026

Non-Gaussian processes and neural networks at finite widths

arXiv 1910.00019 ↗First posted 30 Sep 2019stat.ML cond-mat.dis-nn cs.LGcited by 78

Not on the map. Why?

Abstract (arXiv)

Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors. The methodology developed herein allows us to track the flow of preactivation distributions by progressively integrating out random variables from lower to higher layers, reminiscent of renormalization-group flow. We further develop a perturbative procedure to perform Bayesian inference with weakly non-Gaussian priors.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 16; 2023: 20; 2024: 12; 2025: 8; 2026: 7.

78 papers.

Sort:

  1. 2021Explaining Neural Scaling Lawscited by 363
  2. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  3. 2022A Solvable Model of Neural Scaling Lawscited by 116
  4. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  5. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  6. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  7. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  8. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  9. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  10. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  11. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 43
  12. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
38 more
  1. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  2. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  3. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  4. 2021Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlatorscited by 29
  5. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  6. 2021Nonperturbative renormalization for the neural network-QFT correspondencecited by 24
  7. 2021The edge of chaos: quantum field theory and deep neural networkscited by 24
  8. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  9. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  10. 2023Neural Network Field Theories: Non-Gaussianity, Actions, and Localitycited by 22
  11. 2022Analytic theory for the dynamics of wide quantum neural networkscited by 21
  12. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  13. 2023Quantitative CLTs in Deep Neural Networkscited by 20
  14. 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 19
  15. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  16. 2024Emergent Equivariance in Deep Ensemblescited by 18
  17. 2021A theory of representation learning gives a deep generalisation of kernel methodscited by 15
  18. 2021Rate of Convergence of Polynomial Networks to Gaussian Processescited by 13
  19. 2023Structures of Neural Network Effective Theoriescited by 13
  20. 2022Decomposing neural networks as mappings of correlation functionscited by 12
  21. 2024Coding schemes in neural networks learning classification taskscited by 11
  22. 2022Laziness, Barren Plateau, and Noise in Machine Learningcited by 10
  23. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  24. 2022Wide Bayesian neural networks have a simple weight posterior: theory and accelerated samplingcited by 9
  25. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  26. 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applicationscited by 8
  27. 2023Catapult Dynamics and Phase Transitions in Quadratic Netscited by 8
  28. 2023A Quantitative Functional Central Limit Theorem for Shallow Neural Networkscited by 8
  29. 2021Why is AI hard and Physics simple?cited by 7
  30. 2025Fermions and Supersymmetry in Neural Network Field Theoriescited by 6
  31. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  32. 2023Feature Learning and Generalization in Deep Networks with Orthogonal Weightscited by 5
  33. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
  34. 2024TASI Lectures on Physics for Machine Learningcited by 5
  35. 2021Infinite Neural Network Quantum States: Entanglement and Training Dynamicscited by 4
  36. 2023Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networkscited by 4
  37. 2024Equivariant Neural Tangent Kernelscited by 4
  38. 2025Finite-Width Neural Tangent Kernels from Feynman Diagramscited by 4

50 most cited of 78 All 78 (JSON)