mlsanityData to 14 Oct 2026

On Using Large-Batches in Federated Learning

arXiv 2509.10537 ↗First posted 5 Sep 2025cs.LG cs.AI cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Efficient Federated learning (FL) is crucial for training deep networks over devices with limited compute resources and bounded networks. With the advent of big data, devices either generate or collect multimodal data to train either generic or local-context aware networks, particularly when data privacy and locality is vital. FL algorithms generally trade-off between parallel and statistical performance, improving model quality at the cost of higher communication frequency, or vice versa. Under frequent synchronization settings, FL over a large cluster of devices may perform more work per-training iteration by processing a larger global batch-size, thus attaining considerable training speedup. However, this may result in poor test performance (i.e., low test loss or accuracy) due to generalization degradation issues associated with large-batch training. To address these challenges with large-batches, this work proposes our vision of exploiting the trade-offs between small and large-batch training, and explore new directions to enjoy both the parallel scaling of large-batches and good generalizability of small-batch training. For the same number of iterations, we observe that our proposed large-batch training technique attains about 32.33% and 3.74% higher test accuracy than small-batch training in ResNet50 and VGG11 models respectively.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 5 Sep 2025.

Builds on

32 of 36 references.

References · 32

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  6. 2017Large Batch Training of Convolutional Networkscited by 535
  7. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 409
  8. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  9. 2018An Empirical Model of Large-Batch Trainingcited by 281
  10. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  11. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  12. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
20 more
  1. 2020The Early Phase of Neural Network Trainingcited by 115
  2. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  3. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  4. 2017Critical Learning Periods in Deep Neural Networkscited by 79
  5. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  6. 2018On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descentcited by 25
  7. 2020Extrapolation for Large-batch Training in Deep Learningcited by 25
  8. 2021Learned Gradient Compression for Distributed Deep Learningcited by 20
  9. 2020Accordion: Adaptive Gradient Communication via Critical Learning Regime Identificationcited by 19
  10. 2018Large batch size training of neural networks with adversarial training and second-order informationcited by 18
  11. 2019An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noisecited by 13
  12. 2023Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batchingcited by 12
  13. 2023GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Trainingcited by 10
  14. 2016Distributed Deep Learning Using Synchronous Stochastic Gradient Descentcited by 8
  15. 2023ScaDLES: Scalable Deep Learning over Streaming data at the Edgecited by 8
  16. 2023Accelerating Distributed ML Training via Selective Synchronizationcited by 7
  17. 2023Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Trainingcited by 5
  18. 2023Revisiting LARS for Large Batch Training Generalization of Neural Networkscited by 5
  19. 2025OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusterscited by 5
  20. 2023Flexible Communication for Optimal Distributed Learning over Unpredictable Networkscited by 3

Built on by

Not cited yet.