mlsanityData to 7 Oct 2026

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

arXiv 1904.00962 ↗First posted 1 Apr 2019cs.LG cs.AI cs.CLcited by 671

Not on the map. Why?

Abstract (arXiv)

Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning rates trains ResNet on ImageNet in a few minutes. However, LARS performs poorly for attention models like BERT, indicating that its performance gains are not consistent across tasks. In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches. Using this strategy, we develop a new layerwise adaptive large batch optimization technique called LAMB; we then provide convergence analysis of LAMB as well as LARS, showing convergence to a stationary point in general nonconvex settings. Our empirical results demonstrate the superior performance of LAMB across various tasks such as BERT and ResNet-50 training with very little hyperparameter tuning. In particular, for BERT training, our optimizer enables use of very large batch sizes of 32868 without any degradation of performance. By increasing the batch size to the memory limit of a TPUv3 Pod, BERT training time can be reduced from 3 days to just 76 minutes (Table 1). The LAMB implementation is available at https://github.com/tensorflow/addons/blob/master/tensorflow_addons/optimizers/lamb.py

Key facts. Cited by 671 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 120; 2022: 114; 2023: 154; 2024: 110; 2025: 93; 2026: 79.

671 papers.

Sort:

By area:

  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  6. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  7. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  8. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  9. 2022DeiT III: Revenge of the ViTcited by 417
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  11. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  12. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
38 more
  1. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  2. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  3. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  4. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  7. 2023The Stable Signature: Rooting Watermarks in Latent Diffusion Modelscited by 289
  8. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  9. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  10. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  11. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  12. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  13. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  14. 2022Multi-Game Decision Transformerscited by 212
  15. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 203
  16. 2022Online Decision Transformercited by 194
  17. 2023On the Importance of Noise Scheduling for Diffusion Modelscited by 193
  18. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  19. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  20. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  21. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  22. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  23. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  24. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  25. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  26. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  27. 2022MetaFormer Baselines for Visioncited by 131
  28. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  29. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  30. 2021Large-Scale Differentially Private BERTcited by 117
  31. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  32. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  33. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 112
  34. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  35. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  36. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  37. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  38. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104

50 most cited of 671 All 671 (JSON)