mlsanityData to 7 Oct 2026

Fokker-Planck to Callan-Symanzik: evolution of weight matrices under training

arXiv 2501.09659 ↗First posted 16 Jan 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The dynamical evolution of a neural network during training has been an incredibly fascinating subject of study. First principal derivation of generic evolution of variables in statistical physics systems has proved useful when used to describe training dynamics conceptually, which in practice means numerically solving equations such as Fokker-Planck equation. Simulating entire networks inevitably runs into the curse of dimensionality. In this paper, we utilize Fokker-Planck to simulate the probability density evolution of individual weight matrices in the bottleneck layers of a simple 2-bottleneck-layered auto-encoder and compare the theoretical evolutions against the empirical ones by examining the output data distributions. We also derive physically relevant partial differential equations such as Callan-Symanzik and Kardar-Parisi-Zhang equations from the dynamical equation we have.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

29 of 46 references.

References · 29

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 835
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
  7. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  10. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  11. 2016Tutorial on Variational Autoencoderscited by 404
  12. 2021Conditional Image Generation with Score-Based Diffusion Modelscited by 195
17 more
  1. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  2. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  3. 2024Accelerating Convergence of Score-Based Diffusion Models, Provablycited by 74
  4. 2024Optimal Flow Matching: Learning Straight Trajectories in Just One Stepcited by 74
  5. 2020Neural Networks and Quantum Field Theorycited by 58
  6. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  7. 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 49
  8. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  9. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  10. 2023Renormalizing Diffusion Modelscited by 16
  11. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  12. 2017Langevin Dynamics with Continuous Tempering for Training Deep Neural Networkscited by 7
  13. 2023Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplingscited by 6
  14. 2021Scalable computation of dynamic flow problems via multi-marginal graph-structured optimal transportcited by 5
  15. 2023Machine learning in and out of equilibriumcited by 5
  16. 2022Stochastic Langevin Differential Inclusions with Applications to Machine Learningcited by 2
  17. 2022Langevin algorithms for Markovian Neural Networks and Deep Stochastic controlcited by 2

Built on by

Not cited yet.