Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent
Not on the map. Why?
Abstract (arXiv)
We show that the behavior of stochastic gradient descent is related to Bayesian statistics by showing that SGD is effectively diffusion on a fractal landscape, where the fractal dimension can be accounted for in a purely Bayesian way. By doing this we show that SGD can be regarded as a modified Bayesian sampler which accounts for accessibility constraints induced by the fractal structure of the loss landscape. We verify our results experimentally by examining the diffusion of weights during training. These results offer insight into the factors which determine the learning process, and seemingly answer the question of how SGD and purely Bayesian sampling are related.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 67 references that are in the map. First posted on arXiv 28 Mar 2025.
Builds on
31 of 67 references.
References · 31
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2022In-context Learning and Induction Headscited by 964
- 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
- 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
- 2014The Loss Surfaces of Multilayer Networkscited by 280
- 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
- 2017High-dimensional dynamics of generalization error in neural networkscited by 241
- 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
- 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
19 more
- 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
- 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
- 2021Acquisition of Chess Knowledge in AlphaZerocited by 123
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
- 2020Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networkscited by 62
- 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 57
- 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
- 2020Is SGD a Bayesian sampler? Well, almostcited by 50
- 2020Deep Learning is Singular, and That's Goodcited by 49
- 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 44
- 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
- 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 36
- 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
- 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 29
- 2021The effective noise of Stochastic Gradient Descentcited by 25
- 2021Fractal Structure and Generalization Properties of Stochastic Optimization Algorithmscited by 23
- 2020On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descentcited by 11
- 2020Anomalous diffusion dynamics of learning in deep neural networkscited by 9
- 2024Review and Prospect of Algebraic Research in Equivalent Framework between Statistical Mechanics and Machine Learning Theorycited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.