mlsanityData to 14 Oct 2026

Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)

arXiv 2309.13681 ↗First posted 24 Sep 2023cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

As models for nature language processing (NLP), computer vision (CV) and recommendation systems (RS) require surging computation, a large number of GPUs/TPUs are paralleled as a large batch (LB) to improve training throughput. However, training such LB tasks often meets large generalization gap and downgrades final precision, which limits enlarging the batch size. In this work, we develop the variance reduced gradient descent technique (VRGD) based on the gradient signal to noise ratio (GSNR) and apply it onto popular optimizers such as SGD/Adam/LARS/LAMB. We carry out a theoretical analysis of convergence rate to explain its fast training dynamics, and a generalization analysis to demonstrate its smaller generalization gap on LB training. Comprehensive experiments demonstrate that VRGD can accelerate training (1~ 2 ×), narrow generalization gap and improve final accuracy. We push the batch size limit of BERT pretraining up to 128k/64k and DLRM to 512k without noticeable accuracy loss. We improve ImageNet Top-1 accuracy at 96k by 0.52pp than LARS. The generalization gap of BERT and ImageNet training is significantly reduce by over 65%.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 42 of its 52 references that are in the map. First posted on arXiv 24 Sep 2023.

Builds on

42 of 52 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  11. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  12. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
30 more
  1. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  2. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  3. 2017Large Batch Training of Convolutional Networkscited by 535
  4. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  5. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  6. 2018An Empirical Model of Large-Batch Trainingcited by 281
  7. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  8. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  9. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  10. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  11. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  12. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
  13. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 95
  14. 2021M6: A Chinese Multimodal Pretrainercited by 91
  15. 2018Tighter Variational Bounds are Not Necessarily Bettercited by 76
  16. 2016Variance Reduction for Faster Non-Convex Optimizationcited by 72
  17. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
  18. 2020Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Wellcited by 57
  19. 2020Understanding Why Neural Networks Generalize Well Through GSNR of Parameterscited by 40
  20. 2021DecentLaM: Decentralized Momentum SGD for Large-batch Deep Trainingcited by 40
  21. 2019Augment your batch: better training with larger batchescited by 37
  22. 2017AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networkscited by 29
  23. 2017Reducing Reparameterization Gradient Variancecited by 27
  24. 2020Extrapolation for Large-batch Training in Deep Learningcited by 25
  25. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20
  26. 2021Concurrent Adversarial Learning for Large-Batch Trainingcited by 11
  27. 2020Exploring the limits of Concurrency in ML Training on Google TPUscited by 9
  28. 2020Accelerated Large Batch Optimization of BERT Pretraining in 54 minutescited by 7
  29. 2018Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descentcited by 6
  30. 2020Scaling Distributed Training with Adaptive Summationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Trainingcited by 4
  2. 2024CGLearn: Consistent Gradient-Based Learning for Out-of-Distribution Generalizationcited by 1
  3. 2026GNMR: Runtime Stability Control for Low-Precision Large Language Model Trainingcited by 0