mlsanityData to 7 Oct 2026

Federated Learning with Buffered Asynchronous Aggregation

arXiv 2106.06639 ↗First posted 11 Jun 2021cs.LGcited by 188

Not on the map. Why?

Abstract (arXiv)

Scalability and privacy are two critical concerns for cross-device federated learning (FL) systems. In this work, we identify that synchronous FL - synchronized aggregation of client updates in FL - cannot scale efficiently beyond a few hundred clients training in parallel. It leads to diminishing returns in model performance and training speed, analogous to large-batch training. On the other hand, asynchronous aggregation of client updates in FL (i.e., asynchronous FL) alleviates the scalability issue. However, aggregating individual client updates is incompatible with Secure Aggregation, which could result in an undesirable level of privacy for the system. To address these concerns, we propose a novel buffered asynchronous aggregation method, FedBuff, that is agnostic to the choice of optimizer, and combines the best properties of synchronous and asynchronous FL. We empirically demonstrate that FedBuff is 3.3x more efficient than synchronous FL and up to 2.5x more efficient than asynchronous FL, while being compatible with privacy-preserving technologies such as Secure Aggregation and differential privacy. We provide theoretical convergence guarantees in a smooth non-convex setting. Finally, we show that under differentially private training, FedBuff can outperform FedAvgM at low privacy settings and achieve the same utility for higher privacy settings.

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Builds on 59 of its 88 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

59 of 88 references.

References · 59

  1. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2018Group Normalizationcited by 1,514
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  9. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  10. 2020Adaptive Federated Optimizationcited by 1,036
  11. 2019Deep Leakage from Gradientscited by 1,004
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
38 more
  1. 2019Towards Federated Learning at Scale: System Designcited by 826
  2. 2018LEAF: A Benchmark for Federated Settingscited by 780
  3. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  4. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  5. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  6. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  7. 2017Large Batch Training of Convolutional Networkscited by 529
  8. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 509
  9. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  10. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  11. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  12. 2017Three Factors Influencing Minima in SGDcited by 270
  13. 2019Asynchronous Federated Optimizationcited by 264
  14. 2018Scaling Neural Machine Translationcited by 204
  15. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  16. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  17. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  18. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
  19. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  20. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  21. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  22. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 150
  23. 2017Revisiting Distributed Synchronous SGDcited by 142
  24. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
  25. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
  26. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 98
  27. 2021PPFL: Privacy-preserving Federated Learning with Trusted Execution Environmentscited by 93
  28. 2019SAFA: a Semi-Asynchronous Protocol for Fast Federated Learning with Low Overheadcited by 82
  29. 2020Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learningcited by 78
  30. 2015Speeding Up Distributed Machine Learning Using Codescited by 76
  31. 2018Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGDcited by 68
  32. 2015Perturbed Iterate Analysis for Asynchronous Stochastic Optimizationcited by 67
  33. 2018Lagrange Coded Computing: Optimal Design for Resiliency, Security and Privacycited by 66
  34. 2020Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneitycited by 56
  35. 2017Prochlo: Strong Privacy for Analytics in the Crowdcited by 51
  36. 2021Gradient Disaggregation: Breaking Privacy in Federated Learning by Reconstructing the User Participant Matrixcited by 43
  37. 2020Advances in Asynchronous Parallel and Distributed Optimizationcited by 40
  38. 2020Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluationcited by 39

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 29; 2023: 33; 2024: 50; 2025: 40; 2026: 24.

188 papers.

Sort:

By area:

  1. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 89
  2. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  3. 2021Eluding Secure Aggregation in Federated Learning via Model Inconsistencycited by 66
  4. 2021LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learningcited by 63
  5. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  6. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  7. 2021Federated Learning for Internet of Things: Applications, Challenges, and Opportunitiescited by 49
  8. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  9. 2021Anarchic Federated Learningcited by 42
  10. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  11. 2023LOKI: Large-scale Data Reconstruction Attack against Federated Learning through Model Manipulationcited by 35
  12. 2022Federated Learning Under Intermittent Client Availability and Time-Varying Communication Constraintscited by 30
38 more
  1. 2022AFLGuard: Byzantine-robust Asynchronous Federated Learningcited by 29
  2. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  3. 2022FedSpace: An Efficient Federated Learning Framework at Satellites and Ground Stationscited by 28
  4. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  5. 2024Asynchronous Local-SGD Training for Language Modelingcited by 24
  6. 2022On the (In)security of Peer-to-Peer Decentralized Machine Learningcited by 23
  7. 2022One Model to Unite Them All: Personalized Federated Learning of Multi-Contrast MRI Synthesiscited by 22
  8. 2023FS-Real: Towards Real-World Cross-Device Federated Learningcited by 22
  9. 2022FedHAP: Fast Federated Learning for LEO Constellations Using Collaborative HAPscited by 21
  10. 2024SatFed: A Resource-Efficient LEO Satellite-Assisted Heterogeneous Federated Learning Frameworkcited by 21
  11. 2021CodedPaddedFL and CodedSecAgg: Straggler Mitigation and Secure Aggregation in Federated Learningcited by 20
  12. 2022Scheduling and Aggregation Design for Asynchronous Federated Learning over Wireless Networkscited by 20
  13. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19
  14. 2022AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregationcited by 19
  15. 2024Advances in APPFL: A Comprehensive and Extensible Federated Learning Frameworkcited by 19
  16. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  17. 2022Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneitycited by 17
  18. 2022A General Theory for Federated Optimization with Asynchronous and Heterogeneous Clients Updatescited by 17
  19. 2023FLINT: A Platform for Federated Learning Integrationcited by 17
  20. 2023FedVS: Straggler-Resilient and Privacy-Preserving Vertical Federated Learning for Split Modelscited by 17
  21. 2023Data and Model Poisoning Backdoor Attacks on Wireless Federated Learning, and the Defense Mechanisms: A Comprehensive Surveycited by 17
  22. 2023AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithmscited by 16
  23. 2023On the Role of Server Momentum in Federated Learningcited by 16
  24. 2022Combined Federated and Split Learning in Edge Computing for Ubiquitous Intelligence in Internet of Things: State of the Art and Future Directionscited by 14
  25. 2023FedBone: Towards Large-Scale Federated Multi-Task Learningcited by 14
  26. 2025POPri: Private Federated Learning using Preference-Optimized Synthetic Datacited by 14
  27. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  28. 2023Amplitude-Varying Perturbation for Balancing Privacy and Utility in Federated Learningcited by 13
  29. 2023FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices using a Computing Power Aware Schedulercited by 13
  30. 2021Edge-Native Intelligence for 6G Communications Driven by Federated Learning: A Survey of Trends and Challengescited by 12
  31. 2022Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropoutcited by 12
  32. 2023FLGo: A Fully Customizable Federated Learning Platformcited by 12
  33. 2024FedFa: A Fully Asynchronous Training Paradigm for Federated Learningcited by 12
  34. 2024Advances in Robust Federated Learning: A Survey with Heterogeneity Considerationscited by 12
  35. 2022Unbounded Gradients in Federated Learning with Buffered Asynchronous Aggregationcited by 11
  36. 2023Async-HFL: Efficient and Robust Asynchronous Federated Learning in Hierarchical IoT Networkscited by 11
  37. 2023Asynchronous SGD on Graphs: a Unified Framework for Asynchronous Decentralized and Federated Optimizationcited by 11
  38. 2021Secure Aggregation for Buffered Asynchronous Federated Learningcited by 10

50 most cited of 188 All 188 (JSON)