mlsanityData to 14 Oct 2026

On Large-Cohort Training for Federated Learning

arXiv 2106.07820 ↗First posted 15 Jun 2021cs.LG cs.DCcited by 74

Not on the map. Why?

Abstract (arXiv)

Federated learning methods typically learn a model by iteratively sampling updates from a population of clients. In this work, we explore how the number of clients sampled at each round (the cohort size) impacts the quality of the learned model and the training dynamics of federated learning algorithms. Our work poses three fundamental questions. First, what challenges arise when trying to scale federated learning to larger cohorts? Second, what parallels exist between cohort sizes in federated learning and batch sizes in centralized learning? Last, how can we design federated learning methods that effectively utilize larger cohort sizes? We give partial answers to these questions based on extensive empirical evaluation. Our work highlights a number of challenges stemming from the use of larger cohorts. While some of these (such as generalization issues and diminishing returns) are analogs of large-batch training challenges, others (including training failures and fairness concerns) are unique to federated learning.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Builds on 57 of its 79 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

57 of 79 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2016Deep Learning with Differential Privacycited by 2,728
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  8. 2018Group Normalizationcited by 1,524
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  11. 2020Adaptive Federated Optimizationcited by 1,046
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
38 more
  1. 2019Towards Federated Learning at Scale: System Designcited by 832
  2. 2018LEAF: A Benchmark for Federated Settingscited by 792
  3. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  4. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  5. 2020Federated Learning with Matched Averagingcited by 564
  6. 2017Large Batch Training of Convolutional Networkscited by 535
  7. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  8. 2019Agnostic Federated Learningcited by 456
  9. 2019Fair Resource Allocation in Federated Learningcited by 394
  10. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 352
  11. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  12. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  13. 2018An Empirical Model of Large-Batch Trainingcited by 281
  14. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  15. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  16. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  17. 2019Differentially Private Learning with Adaptive Clippingcited by 220
  18. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  19. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 198
  20. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  21. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  22. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  23. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  24. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  25. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  26. 2020Optimal Client Sampling for Federated Learningcited by 124
  27. 2018Distributed Differential Privacy via Shufflingcited by 115
  28. 2018Convergence of Gradient Descent on Separable Datacited by 110
  29. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  30. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 102
  31. 2021Federated Evaluation and Tuning for On-Device Personalization: System Design & Applicationscited by 85
  32. 2019First Analysis of Local GD on Heterogeneous Datacited by 81
  33. 2015Speeding Up Distributed Machine Learning Using Codescited by 76
  34. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  35. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  36. 2019Active Federated Learningcited by 63
  37. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  38. 2017Prochlo: Strong Privacy for Analytics in the Crowdcited by 51

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 17; 2023: 21; 2024: 23; 2025: 4; 2026: 2.

74 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 315
  2. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  3. 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 125
  4. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  5. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 90
  6. 2022Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streamscited by 59
  7. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 58
  8. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  9. 2021Anarchic Federated Learningcited by 42
  10. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 38
  11. 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
  12. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
38 more
  1. 2022Federated Learning Under Intermittent Client Availability and Time-Varying Communication Constraintscited by 30
  2. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  3. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  4. 2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 27
  5. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  6. 2024Federated LoRA with Sparse Communicationcited by 26
  7. 2022Server-Side Stepsizes and Sampling Without Replacement Provably Help in Federated Optimizationcited by 23
  8. 2023Topology-aware Federated Learning in Edge Computing: A Comprehensive Surveycited by 23
  9. 2023FLINT: A Platform for Federated Learning Integrationcited by 17
  10. 2024FedImpro: Measuring and Improving Client Update in Federated Learningcited by 15
  11. 2022Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theorycited by 14
  12. 2023DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency for Federated Learning with Static and Streaming Datasetcited by 14
  13. 2024The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communicationcited by 12
  14. 2021FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learningcited by 11
  15. 2022Federated Select: A Primitive for Communication- and Memory-Efficient Federated Learningcited by 11
  16. 2023QuickDrop: Efficient Federated Unlearning by Integrated Dataset Distillationcited by 11
  17. 2022Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learningcited by 10
  18. 2024Not All Federated Learning Algorithms Are Created Equal: A Performance Evaluation Studycited by 10
  19. 2024Tackling Data Heterogeneity in Federated Learning via Loss Decompositioncited by 9
  20. 2021AdaSplit: Adaptive Trade-offs for Resource-constrained Distributed Deep Learningcited by 8
  21. 2022FLAME: Federated Learning Across Multi-device Environmentscited by 8
  22. 2023Making Batch Normalization Great in Federated Deep Learningcited by 8
  23. 2023Green Federated Learningcited by 8
  24. 2023Improving Accelerated Federated Learning with Compression and Importance Samplingcited by 8
  25. 2023Federated Learning with Manifold Regularization and Normalized Update Reaggregationcited by 8
  26. 2024Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuningcited by 8
  27. 2022QUIC-FL: Quick Unbiased Compression for Federated Learningcited by 7
  28. 2022On Noisy Evaluation in Federated Hyperparameter Tuningcited by 7
  29. 2024Federated Unsupervised Domain Generalization using Global and Local Alignment of Gradientscited by 7
  30. 2024Version age-based client scheduling policy for federated learningcited by 5
  31. 2024Worldwide Federated Training of Language Modelscited by 5
  32. 2024FedStaleWeight: Buffered Asynchronous Federated Learning with Fair Aggregation via Staleness Reweightingcited by 5
  33. 2022Reconciling Security and Communication Efficiency in Federated Learningcited by 4
  34. 2023Asynchronous Wireless Federated Learning with Probabilistic Client Selectioncited by 4
  35. 2022Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compressioncited by 3
  36. 2023A Comparative Study of Federated Learning Models for COVID-19 Detectioncited by 3
  37. 2023FLEdge: Benchmarking Federated Machine Learning Applications in Edge Computing Systemscited by 3
  38. 2023Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learningcited by 3

50 most cited of 74 All 74 (JSON)