# On the Dynamics of Inference and Learning

arXiv: https://arxiv.org/abs/2204.12939 · HTML: https://mlsanity.com/paper/2204.12939/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 40 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Principles of Deep Learning Theory](https://mlsanity.com/paper/2106.10165/) (2021)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [Neural Networks and Quantum Field Theory](https://mlsanity.com/paper/2008.08601/) (2020)
- [Building Quantum Field Theories Out of Neurons](https://mlsanity.com/paper/2112.04527/) (2021)
- [Quantum field-theoretic machine learning](https://mlsanity.com/paper/2102.09449/) (2021)
- [Baryons from Mesons: A Machine Learning Perspective](https://mlsanity.com/paper/2003.10445/) (2020)
- [Machine learning with quantum field theories](https://mlsanity.com/paper/2109.07730/) (2021)
- [Quantum field theories, Markov random fields and machine learning](https://mlsanity.com/paper/2110.10928/) (2021)

## Built on by (most cited first, 9 of 9)

- [FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information](https://mlsanity.com/paper/2405.12807/) (2024)
- [Bayesian Renormalization](https://mlsanity.com/paper/2305.10491/) (2023)
- [The Inverse of Exact Renormalization Group Flows as Statistical Inference](https://mlsanity.com/paper/2212.11379/) (2022)
- [Hierarchical Neural Simulation-Based Inference Over Event Ensembles](https://mlsanity.com/paper/2306.12584/) (2023)
- [Bayesian RG Flow in Neural Network Field Theories](https://mlsanity.com/paper/2405.17538/) (2024)
- [Wilsonian Renormalization of Neural Network Gaussian Processes](https://mlsanity.com/paper/2405.06008/) (2024)
- [Grokking vs. Learning: Same Features, Different Encodings](https://mlsanity.com/paper/2502.01739/) (2025)
- [Conjugate Natural Selection](https://mlsanity.com/paper/2208.13898/) (2022)
- [Optimal Pruning for Neural Architectures using Fisher Information Distances](https://mlsanity.com/paper/2609.16129/) (2026)

## Abstract (arXiv)

Statistical Inference is the process of determining a probability distribution over the space of parameters of a model given a data set. As more data becomes available this probability distribution becomes updated via the application of Bayes' theorem. We present a treatment of this Bayesian updating process as a continuous dynamical system. Statistical inference is then governed by a first order differential equation describing a trajectory or flow in the information geometry determined by a parametric family of models. We solve this equation for some simple models and show that when the Cram'er-Rao bound is saturated the learning rate is governed by a simple 1/T power-law, with T a time-like variable denoting the quantity of data. The presence of hidden variables can be incorporated in this setting, leading to an additional driving term in the resulting flow equation. We illustrate this with both analytic and numerical examples based on Gaussians and Gaussian Random Processes and inference of the coupling constant in the 1D Ising model. Finally we compare the qualitative behaviour exhibited by Bayesian flows to the training of various neural networks on benchmarked data sets such as MNIST and CIFAR10 and show how that for networks exhibiting small final losses the simple power-law is also satisfied.