# On Using Large-Batches in Federated Learning

arXiv: https://arxiv.org/abs/2509.10537 · HTML: https://mlsanity.com/paper/2509.10537/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 36 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Builds on (most cited first, 30 of 32)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [An Empirical Model of Large-Batch Training](https://mlsanity.com/paper/1812.06162/) (2018)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Don't Use Large Mini-Batches, Use Local SGD](https://mlsanity.com/paper/1808.07217/) (2018)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [Hessian-based Analysis of Large Batch Training and Robustness to Adversaries](https://mlsanity.com/paper/1802.08241/) (2018)
- [Decentralized Federated Learning: A Survey and Perspective](https://mlsanity.com/paper/2306.01603/) (2023)
- [Critical Learning Periods in Deep Neural Networks](https://mlsanity.com/paper/1711.08856/) (2017)
- [Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes](https://mlsanity.com/paper/1807.11205/) (2018)
- [On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent](https://mlsanity.com/paper/1811.12941/) (2018)
- [Extrapolation for Large-batch Training in Deep Learning](https://mlsanity.com/paper/2006.05720/) (2020)
- [Learned Gradient Compression for Distributed Deep Learning](https://mlsanity.com/paper/2103.08870/) (2021)
- [Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification](https://mlsanity.com/paper/2010.16248/) (2020)
- [Large batch size training of neural networks with adversarial training and second-order information](https://mlsanity.com/paper/1810.01021/) (2018)
- [An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise](https://mlsanity.com/paper/1902.08234/) (2019)
- [Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching](https://mlsanity.com/paper/2305.12213/) (2023)
- [GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training](https://mlsanity.com/paper/2305.12201/) (2023)
- [Distributed Deep Learning Using Synchronous Stochastic Gradient Descent](https://mlsanity.com/paper/1602.06709/) (2016)
- [ScaDLES: Scalable Deep Learning over Streaming data at the Edge](https://mlsanity.com/paper/2301.08897/) (2023)
- [Accelerating Distributed ML Training via Selective Synchronization](https://mlsanity.com/paper/2307.07950/) (2023)
- [Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Training](https://mlsanity.com/paper/2303.06659/) (2023)
- [Revisiting LARS for Large Batch Training Generalization of Neural Networks](https://mlsanity.com/paper/2309.14053/) (2023)

## Abstract (arXiv)

Efficient Federated learning (FL) is crucial for training deep networks over devices with limited compute resources and bounded networks. With the advent of big data, devices either generate or collect multimodal data to train either generic or local-context aware networks, particularly when data privacy and locality is vital. FL algorithms generally trade-off between parallel and statistical performance, improving model quality at the cost of higher communication frequency, or vice versa. Under frequent synchronization settings, FL over a large cluster of devices may perform more work per-training iteration by processing a larger global batch-size, thus attaining considerable training speedup. However, this may result in poor test performance (i.e., low test loss or accuracy) due to generalization degradation issues associated with large-batch training. To address these challenges with large-batches, this work proposes our vision of exploiting the trade-offs between small and large-batch training, and explore new directions to enjoy both the parallel scaling of large-batches and good generalizability of small-batch training. For the same number of iterations, we observe that our proposed large-batch training technique attains about 32.33% and 3.74% higher test accuracy than small-batch training in ResNet50 and VGG11 models respectively.