mlsanityData to 14 Oct 2026

On Large Batch Training and Sharp Minima: A Fokker-Planck Perspective

arXiv 2112.00987 ↗First posted 2 Dec 2021cs.LG math.ST stat.THcited by 5

Not on the map. Why?

Abstract (arXiv)

We study the statistical properties of the dynamic trajectory of stochastic gradient descent (SGD). We approximate the mini-batch SGD and the momentum SGD as stochastic differential equations (SDEs). We exploit the continuous formulation of SDE and the theory of Fokker-Planck equations to develop new results on the escaping phenomenon and the relationship with large batch and sharp minima. In particular, we find that the stochastic process solution tends to converge to flatter minima regardless of the batch size in the asymptotic regime. However, the convergence rate is rigorously proven to depend on the batch size. These results are validated empirically with various datasets and models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 15 of its 33 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

15 of 33 references.

References · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  7. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  8. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  9. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  10. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  11. 2017Three Factors Influencing Minima in SGDcited by 273
  12. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
3 more
  1. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
  2. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  3. 2017Deep Relaxation: partial differential equations for optimizing deep neural networkscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 2; 2026: 2.

5 papers.

  1. 2023Privacy Risk for anisotropic Langevin dynamics using relative entropy boundscited by 2
  2. 2025Model Merging via Multi-Teacher Knowledge Distillationcited by 2
  3. 2025Optimal Condition for Initialization Variance in Deep Neural Networks: An SGD Dynamics Perspectivecited by 0
  4. 2026Nonreversible Gauge Fields in Fokker--Planck Dynamics: Supersymmetric Hamiltonians and Learned Finite Forcescited by 0
  5. 2026Avoiding unsafe sets when training with Langevin Dynamicscited by 0