mlsanityData to 7 Oct 2026

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

arXiv 1609.04836 ↗First posted 15 Sep 2016cs.LG math.OCcited by 1,337

Not on the map. Why?

Abstract (arXiv)

The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say 32-512 data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.

Key facts. Cited by 1,337 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 205; 2022: 215; 2023: 245; 2024: 230; 2025: 222; 2026: 219.

1,337 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  3. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  4. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  5. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  7. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  8. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  9. 2022How Do Vision Transformers Work?cited by 305
  10. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  11. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  12. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
38 more
  1. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  2. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  3. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  4. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  5. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 175
  6. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  7. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  8. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  9. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  10. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  11. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  12. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  13. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  14. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  15. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  16. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  17. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  18. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  19. 2023On Memorization in Diffusion Modelscited by 122
  20. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  21. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
  22. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  23. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  24. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  25. 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 110
  26. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  27. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  28. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  29. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  30. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  31. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
  32. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  33. 2021The Bayesian Learning Rulecited by 95
  34. 2022Sharpness-Aware Training for Freecited by 95
  35. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 93
  36. 2022Vision Transformers provably learn spatial structurecited by 92
  37. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  38. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 89

50 most cited of 1,337 All 1,337 (JSON)