mlsanityData to 7 Oct 2026

Wilsonian Renormalization of Neural Network Gaussian Processes

arXiv 2405.06008 ↗First posted 9 May 2024cs.LG cond-mat.dis-nn hep-thcited by 8

Not on the map. Why?

Abstract (arXiv)

Separating relevant and irrelevant information is key to any modeling process or scientific inquiry. Theoretical physics offers a powerful tool for achieving this in the form of the renormalization group (RG). Here we demonstrate a practical approach to performing Wilsonian RG in the context of Gaussian Process (GP) Regression. We systematically integrate out the unlearnable modes of the GP kernel, thereby obtaining an RG flow of the GP in which the data sets the IR scale. In simple cases, this results in a universal flow of the ridge parameter, which becomes input-dependent in the richer scenario in which non-Gaussianities are included. In addition to being analytically tractable, this approach goes beyond structural analogies between RG and neural networks by providing a natural connection between RG flow and learnable vs. unlearnable modes. Studying such flows may improve our understanding of feature learning in deep neural networks, and enable us to identify potential universality classes in these models.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 34 of its 58 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

34 of 58 references.

References · 34

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2017Deep Neural Networks as Gaussian Processescited by 617
  5. 2021Explaining Neural Scaling Lawscited by 353
  6. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  7. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  8. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  9. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  10. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  11. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  12. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
22 more
  1. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  2. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  3. 2020Neural Networks and Quantum Field Theorycited by 58
  4. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  5. 2014An exact mapping between the Variational Renormalization Group and Deep Learningcited by 52
  6. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  7. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  8. 2017Mutual Information, Neural Networks and the Renormalization Groupcited by 32
  9. 2020Predicting the outputs of finite deep neural networks trained with noisy gradientscited by 31
  10. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  11. 2021Nonperturbative renormalization for the neural network-QFT correspondencecited by 24
  12. 2021The edge of chaos: quantum field theory and deep neural networkscited by 23
  13. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  14. 2023Renormalizing Diffusion Modelscited by 16
  15. 2023Bayesian Renormalizationcited by 13
  16. 2021Towards quantifying information flows: relative entropy in deep neural networks and the renormalization groupcited by 12
  17. 2022Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional datacited by 12
  18. 2022The Inverse of Exact Renormalization Group Flows as Statistical Inferencecited by 11
  19. 2022On the Dynamics of Inference and Learningcited by 9
  20. 2016PCA meets RGcited by 8
  21. 2022Renormalization in the neural network-quantum field theory correspondencecited by 7
  22. 2024NCoder -- A Quantum Field Theory approach to encoding datacited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 3.

8 papers.

  1. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  2. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  3. 2025Fermions and Supersymmetry in Neural Network Field Theoriescited by 5
  4. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  5. 2024When narrower is better: the narrow width limit of Bayesian parallel branching neural networkscited by 1
  6. 2025A simple mean field model of feature learningcited by 0
  7. 2026Deep neural networks as lattice gauge theoriescited by 0
  8. 2026What Neural Network Field Theory Can and Cannot Realise on a Computercited by 0