mlsanityData to 14 Oct 2026

Large Batch Training of Convolutional Networks

arXiv 1708.03888 ↗First posted 13 Aug 2017cs.CVcited by 535

Not on the map. Why?

Abstract (arXiv)

A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. But training with large batch size often results in the lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome this optimization difficulties we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy.

Key facts. Cited by 535 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 105; 2022: 146; 2023: 103; 2024: 69; 2025: 60; 2026: 52.

535 papers.

Sort:

By area:

  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  5. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  6. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  7. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  8. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  9. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  10. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  11. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  12. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
38 more
  1. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  2. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  3. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  4. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  5. 2023A Cookbook of Self-Supervised Learningcited by 250
  6. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  7. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  8. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 198
  9. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  10. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  11. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  12. 2022A Self-Supervised Descriptor for Image Copy Detectioncited by 176
  13. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  14. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  15. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  16. 2022On-Device Training Under 256KB Memorycited by 160
  17. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 160
  18. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  19. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  20. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 150
  21. 2021Self-Supervised Models are Continual Learnerscited by 149
  22. 2021Decoupled Contrastive Learningcited by 143
  23. 2022Training Data Influence Analysis and Estimation: A Surveycited by 135
  24. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  25. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  26. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  27. 2021Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samplescited by 114
  28. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  29. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  30. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  31. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  32. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  33. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  34. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 82
  35. 2021Weakly Supervised Contrastive Learningcited by 81
  36. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  37. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
  38. 2022Object discovery and representation networkscited by 80

50 most cited of 535 All 535 (JSON)