mlsanityData to 14 Oct 2026

Revisiting LARS for Large Batch Training Generalization of Neural Networks

arXiv 2309.14053 ↗First posted 25 Sep 2023cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper explores Large Batch Training techniques using layer-wise adaptive scaling ratio (LARS) across diverse settings, uncovering insights. LARS algorithms with warm-up tend to be trapped in sharp minimizers early on due to redundant ratio scaling. Additionally, a fixed steep decline in the latter phase restricts deep neural networks from effectively navigating early-phase sharp minimizers. Building on these findings, we propose Time Varying LARS (TVLARS), a novel algorithm that replaces warm-up with a configurable sigmoid-like function for robust training in the initial phase. TVLARS promotes gradient exploration early on, surpassing sharp optimizers and gradually transitioning to LARS for robustness in later phases. Extensive experiments demonstrate that TVLARS consistently outperforms LARS and LAMB in most cases, with up to 2% improvement in classification scenarios. Notably, in all self-supervised learning cases, TVLARS dominates LARS and LAMB with performance improvements of up to 10%.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 37 of its 61 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

37 of 61 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2023Segment Anythingcited by 8,234
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  11. 2019Searching for MobileNetV3cited by 1,994
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
25 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  2. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  5. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  8. 2017Large Batch Training of Convolutional Networkscited by 535
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  10. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 288
  11. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  12. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  13. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  14. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  15. 2018Decorrelated Batch Normalizationcited by 83
  16. 2017MegDet: A Large Mini-Batch Object Detectorcited by 66
  17. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  18. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  19. 2020A Survey on Large-scale Machine Learningcited by 35
  20. 2020Cross-Iteration Batch Normalizationcited by 20
  21. 2019Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 secondscited by 12
  22. 2021Automated Learning Rate Scheduler for Large-batch Trainingcited by 10
  23. 2017Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to traincited by 8
  24. 2022Large-batch Optimization for Dense Visual Predictionscited by 4
  25. 2020Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clippingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  2. 2025On Using Large-Batches in Federated Learningcited by 0
  3. 2025D2D Power Allocation via Quantum Graph Neural Networkcited by 0
  4. 2025BertsWin: Resolving Topological Sparsity in 3D Masked Autoencoders via Component-Balanced Structural Optimizationcited by 0
  5. 2026On Privacy in Data-Space Tabular Diffusion Models: Influential Factors, Attacker Knowledge, and Metricscited by 0