mlsanityData to 7 Oct 2026

Don't Decay the Learning Rate, Increase the Batch Size

arXiv 1711.00489 ↗First posted 1 Nov 2017cs.LG cs.CV cs.DCcited by 313

Not on the map. Why?

Abstract (arXiv)

It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reaches equivalent test accuracies after the same number of training epochs, but with fewer parameter updates, leading to greater parallelism and shorter training times. We can further reduce the number of parameter updates by increasing the learning rate ε and scaling the batch size B propto ε. Finally, one can increase the momentum coefficient m and scale B propto 1/(1-m), although this tends to slightly reduce the test accuracy. Crucially, our techniques allow us to repurpose existing training schedules for large batch training with no hyper-parameter tuning. We train ResNet-50 on ImageNet to 76.1% validation accuracy in under 30 minutes.

Key facts. Cited by 313 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 61; 2022: 58; 2023: 56; 2024: 46; 2025: 55; 2026: 37.

313 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  4. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  5. 2023Challenges and Applications of Large Language Modelscited by 251
  6. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  7. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  8. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  9. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  10. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  11. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  12. 2021Deep Learning Through the Lens of Example Difficultycited by 139
38 more
  1. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  2. 2022Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?cited by 128
  3. 2023End-to-End Speech Recognition: A Surveycited by 127
  4. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  5. 2021Large-Scale Differentially Private BERTcited by 117
  6. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  7. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  8. 2022Vision Transformers provably learn spatial structurecited by 92
  9. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  10. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 78
  11. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  12. 2021On Large-Cohort Training for Federated Learningcited by 73
  13. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  14. 2023Separable Physics-Informed Neural Networkscited by 64
  15. 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networkscited by 64
  16. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  17. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  18. 2021Rethinking "Batch" in BatchNormcited by 44
  19. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  20. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  21. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  22. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  23. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  24. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  25. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  26. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 36
  27. 2023Deep networks for system identification: a Surveycited by 35
  28. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 35
  29. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  30. 2021Batch size-invariance for policy optimizationcited by 33
  31. 2023Scaling Laws for Associative Memoriescited by 30
  32. 2024Scaling Law with Learning Rate Annealingcited by 29
  33. 2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
  34. 2022Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?cited by 28
  35. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  36. 2022Aspects of scaling and scalability for flow-based sampling of lattice QCDcited by 26
  37. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  38. 2024Scaling physics-informed hard constraints with mixture-of-expertscited by 24

50 most cited of 313 All 313 (JSON)