mlsanityData to 14 Oct 2026

Stochastic Training is Not Necessary for Generalization

arXiv 2109.14119 ↗First posted 29 Sep 2021cs.LG math.OCcited by 63

Not on the map. Why?

Abstract (arXiv)

It is widely believed that the implicit regularization of SGD is fundamental to the impressive generalization behavior we observe in neural networks. In this work, we demonstrate that non-stochastic full-batch training can achieve comparably strong performance to SGD on CIFAR-10 using modern architectures. To this end, we show that the implicit regularization of SGD can be completely replaced with explicit regularization even when comparing against a strong and well-researched baseline. Our observations indicate that the perceived difficulty of full-batch training may be the result of its optimization properties and the disproportionate time and effort spent by the ML community tuning optimizers and hyperparameters for small-batch training.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 74 of its 112 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

74 of 112 references.

References · 74

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  7. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  8. 2018Group Normalizationcited by 1,524
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  10. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  11. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  12. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
38 more
  1. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  2. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  3. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  4. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  5. 2017Large Batch Training of Convolutional Networkscited by 535
  6. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  7. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  8. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  10. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  11. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  12. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  13. 2018An Empirical Model of Large-Batch Trainingcited by 281
  14. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  15. 2014cuDNN: Efficient Primitives for Deep Learningcited by 245
  16. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  17. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  18. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  19. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  20. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  21. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  22. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 173
  23. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  24. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  25. 2020Implicit Gradient Regularizationcited by 146
  26. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  27. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  28. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  29. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  30. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  31. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 119
  32. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  33. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
  34. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  35. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 102
  36. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  37. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  38. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 18; 2023: 18; 2024: 8; 2025: 12; 2026: 3.

63 papers.

Sort:

By area:

  1. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  2. 2022When Do Flat Minima Optimizers Work?cited by 78
  3. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 76
  4. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  5. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  6. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  7. 2025Deep Learning is Not So Mysterious or Differentcited by 56
  8. 2022Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)cited by 48
  9. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  10. 2021Subspace Adversarial Trainingcited by 41
  11. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
  12. 2022How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularizationcited by 40
38 more
  1. 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queriescited by 40
  2. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  3. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  4. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  5. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  6. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  7. 2022Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GDcited by 19
  8. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  9. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  10. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 12
  11. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  12. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  13. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  14. 2023Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learningcited by 10
  15. 2022Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descentcited by 9
  16. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 9
  17. 2022On the Benefits of Large Learning Rates for Kernel Methodscited by 8
  18. 2022Deep Double Descent via Smooth Interpolationcited by 8
  19. 2024Enhancing Domain Adaptation through Prompt Gradient Alignmentcited by 8
  20. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  21. 2024Dynamical stability and chaos in artificial neural network trajectories along trainingcited by 6
  22. 2021Depth Without the Magic: Inductive Bias of Natural Gradient Descentcited by 5
  23. 2023Select without Fear: Almost All Mini-Batch Schedules Generalize Optimallycited by 5
  24. 2023DropCompute: simple and more robust distributed synchronous training via compute variance reductioncited by 4
  25. 2023Perspectives on the State and Future of Deep Learning - 2023cited by 4
  26. 2024The Impact of Geometric Complexity on Neural Collapse in Transfer Learningcited by 4
  27. 2025SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Trainingcited by 4
  28. 2021Free Probability for predicting the performance of feed-forward fully connected neural networkscited by 3
  29. 2024Just How Flexible are Neural Networks in Practice?cited by 3
  30. 2022Disentangling the Mechanisms Behind Implicit Regularization in SGDcited by 2
  31. 2023U-Clip: On-Average Unbiased Stochastic Gradient Clippingcited by 2
  32. 2023Anti-Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances in Flat Directionscited by 2
  33. 2023Implicit regularisation in stochastic gradient descent: from single-objective to two-player gamescited by 2
  34. 2024Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networkscited by 2
  35. 2024Can Optimization Trajectories Explain Multi-Task Transfer?cited by 2
  36. 2025Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processescited by 2
  37. 2022Learning inducing points and uncertainty on molecular data by scalable variational Gaussian processescited by 1
  38. 2023Differentiable Outlier Detection Enable Robust Deep Multimodal Analysiscited by 1

50 most cited of 63 All 63 (JSON)