mlsanityData to 14 Oct 2026

Bayesian RG Flow in Neural Network Field Theories

arXiv 2405.17538 ↗First posted 27 May 2024hep-th cond-mat.dis-nn cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

The Neural Network Field Theory correspondence (NNFT) is a mapping from neural network (NN) architectures into the space of statistical field theories (SFTs). The Bayesian renormalization group (BRG) is an information-theoretic coarse graining scheme that generalizes the principles of the exact renormalization group (ERG) to arbitrarily parameterized probability distributions, including those of NNs. In BRG, coarse graining is performed in parameter space with respect to an information-theoretic distinguishability scale set by the Fisher information metric. In this paper, we unify NNFT and BRG to form a powerful new framework for exploring the space of NNs and SFTs, which we coin BRG-NNFT. With BRG-NNFT, NN training dynamics can be interpreted as inducing a flow in the space of SFTs from the information-theoretic `IR' rightarrow `UV'. Conversely, applying an information-shell coarse graining to the trained network's parameters induces a flow in the space of SFTs from the information-theoretic `UV' rightarrow `IR'. When the information-theoretic cutoff scale coincides with a standard momentum scale, BRG is equivalent to ERG. We demonstrate the BRG-NNFT correspondence on two analytically tractable examples. First, we construct BRG flows for trained, infinite-width NNs, of arbitrary depth, with generic activation functions. As a special case, we then restrict to architectures with a single infinitely-wide layer, scalar outputs, and generalized cos-net activations. In this case, we show that BRG coarse-graining corresponds exactly to the momentum-shell ERG flow of a free scalar SFT. Our analytic results are corroborated by a numerical experiment in which an ensemble of asymptotically wide NNs are trained and subsequently renormalized using an information-shell BRG scheme.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 72 of its 113 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

72 of 113 references.

References · 72

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2015Variational Inference with Normalizing Flowscited by 2,170
  5. 2016Variational Inference: A Review for Statisticianscited by 1,472
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  8. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  9. 2017Deep Neural Networks as Gaussian Processescited by 622
  10. 2021Explaining Neural Scaling Lawscited by 363
  11. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 338
  12. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
38 more
  1. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  2. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  3. 2016Deep Information Propagationcited by 195
  4. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  5. 2021The Principles of Deep Learning Theorycited by 173
  6. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  7. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  8. 2022A Solvable Model of Neural Scaling Lawscited by 116
  9. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  10. 2023Grokking modular arithmeticcited by 83
  11. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  12. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  13. 2018An elementary introduction to information geometrycited by 76
  14. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  15. 2020Neural Networks and Quantum Field Theorycited by 61
  16. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  17. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  18. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  19. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 43
  20. 2022Learning Lattice Quantum Field Theories with Equivariant Continuous Flowscited by 37
  21. 2017Mutual Information, Neural Networks and the Renormalization Groupcited by 33
  22. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  23. 2020Exact posterior distributions of wide Bayesian neural networkscited by 30
  24. 2021Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlatorscited by 29
  25. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  26. 2021Building Quantum Field Theories Out of Neuronscited by 28
  27. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  28. 2022Aspects of scaling and scalability for flow-based sampling of lattice QCDcited by 26
  29. 2021Nonperturbative renormalization for the neural network-QFT correspondencecited by 24
  30. 2021The edge of chaos: quantum field theory and deep neural networkscited by 24
  31. 2021Scaling Up Machine Learning For Quantum Field Theory with Equivariant Continuous Flowscited by 24
  32. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  33. 2022Bayesian Learning for Neural Networks: an algorithmic surveycited by 23
  34. 2023Normalizing flows for lattice gauge theory in arbitrary space-time dimensioncited by 23
  35. 2023Advances in machine-learning-based sampling motivated by lattice quantum chromodynamicscited by 23
  36. 2019Finite size corrections for neural network Gaussian processescited by 22
  37. 2023Neural Network Field Theories: Non-Gaussianity, Actions, and Localitycited by 22
  38. 2022Bayesian Interpolation with Deep Linear Networkscited by 21

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 3.

9 papers.

  1. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  2. 2025Fermions and Supersymmetry in Neural Network Field Theoriescited by 6
  3. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  4. 2025Grokking vs. Learning: Same Features, Different Encodingscited by 2
  5. 2025Viability of perturbative expansion for quantum field theories on neuronscited by 2
  6. 2025Fokker-Planck to Callan-Symanzik: evolution of weight matrices under trainingcited by 0
  7. 2025Bulk-boundary decomposition of neural networkscited by 0
  8. 2026What Neural Network Field Theory Can and Cannot Realise on a Computercited by 0
  9. 2026Optimal Pruning for Neural Architectures using Fisher Information Distancescited by 0