mlsanityData to 7 Oct 2026

FedScale: Benchmarking Model and System Performance of Federated Learning at Scale

arXiv 2105.11367 ↗First posted 24 May 2021cs.LG cs.AI cs.DCcited by 135

Not on the map. Why?

Abstract (arXiv)

We present FedScale, a federated learning (FL) benchmarking suite with realistic datasets and a scalable runtime to enable reproducible FL research. FedScale datasets encompass a wide range of critical FL tasks, ranging from image classification and object detection to language modeling and speech recognition. Each dataset comes with a unified evaluation protocol using real-world data splits and evaluation metrics. To reproduce realistic FL behavior, FedScale contains a scalable and extensible runtime. It provides high-level APIs to implement FL algorithms, deploy them at scale across diverse hardware and software backends, and evaluate them at scale, all with minimal developer efforts. We combine the two to perform systematic benchmarking experiments and highlight potential opportunities for heterogeneity-aware co-optimizations in FL. FedScale is open-source and actively maintained by contributors from different institutions at http://fedscale.ai. We welcome feedback and contributions from the community.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Builds on 36 of its 60 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

36 of 60 references.

References · 36

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2020Adaptive Federated Optimizationcited by 1,036
  6. 2019Towards Federated Learning at Scale: System Designcited by 826
  7. 2018LEAF: A Benchmark for Federated Settingscited by 780
  8. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  9. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  10. 2015Image-based Recommendations on Styles and Substitutescited by 702
  11. 2018CoQA: A Conversational Question Answering Challengecited by 674
  12. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
24 more
  1. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  2. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  3. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  4. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  5. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  6. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  7. 2020Attack of the Tails: Yes, You Really Can Backdoor Federated Learningcited by 307
  8. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  9. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  10. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  11. 2019Can You Really Backdoor Federated Learning?cited by 250
  12. 2017EMNIST: an extension of MNIST to handwritten letterscited by 212
  13. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  14. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 178
  15. 2019Privacy-preserving Federated Brain Tumour Segmentationcited by 169
  16. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  17. 2019MLPerf Training Benchmarkcited by 114
  18. 2021Federated Evaluation and Tuning for On-Device Personalization: System Design & Applicationscited by 84
  19. 2019AI Benchmark: All About Deep Learning on Smartphones in 2019cited by 76
  20. 2020IBM Federated Learning: an Enterprise Framework White Paper V0.1cited by 57
  21. 2019Federated Learning Of Out-Of-Vocabulary Wordscited by 51
  22. 2017From Lifestyle Vlogs to Everyday Interactionscited by 42
  23. 2019Semi-Cyclic Stochastic Gradient Descentcited by 40
  24. 2020The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systemscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 25; 2023: 27; 2024: 35; 2025: 26; 2026: 18.

135 papers.

Sort:

By area:

  1. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  2. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  3. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  4. 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 121
  5. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  6. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  7. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  8. 2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62
  9. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  10. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  11. 2023Recent Advances on Federated Learning: A Systematic Surveycited by 40
  12. 2022Test-Time Robust Personalization for Federated Learningcited by 39
38 more
  1. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  2. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  3. 2021Resource-Efficient Federated Learningcited by 26
  4. 2024Federated LoRA with Sparse Communicationcited by 26
  5. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
  6. 2022Federated Learning with Position-Aware Neuronscited by 24
  7. 2021Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learningcited by 23
  8. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  9. 2023FS-Real: Towards Real-World Cross-Device Federated Learningcited by 22
  10. 2023FedSecurity: Benchmarking Attacks and Defenses in Federated Learning and Federated LLMscited by 19
  11. 2024Advances in APPFL: A Comprehensive and Extensible Federated Learning Frameworkcited by 19
  12. 2023Decentralized Learning Made Easy with DecentralizePycited by 18
  13. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
  14. 2022Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneitycited by 17
  15. 2023FLINT: A Platform for Federated Learning Integrationcited by 17
  16. 2023Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Uglycited by 17
  17. 2024Collaborative Learning of Anomalies with Privacy (CLAP) for Unsupervised Video Anomaly Detection: A New Baselinecited by 17
  18. 2023A Survey for Federated Learning Evaluations: Goals and Measurescited by 16
  19. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  20. 2024From Challenges and Pitfalls to Recommendations and Opportunities: Implementing Federated Learning in Healthcarecited by 16
  21. 2022Federated Few-Shot Learning for Mobile NLPcited by 15
  22. 2024FedImpro: Measuring and Improving Client Update in Federated Learningcited by 15
  23. 2025Do We Really Need to Design New Byzantine-robust Aggregation Rules?cited by 13
  24. 2023Federated Learning without Full Labels: A Surveycited by 12
  25. 2023FLGo: A Fully Customizable Federated Learning Platformcited by 12
  26. 2024Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Modelscited by 12
  27. 2024One-shot Federated Learning via Synthetic Distiller-Distillate Communicationcited by 12
  28. 2022Towards Energy-Aware Federated Learning on Battery-Powered Clientscited by 11
  29. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuningcited by 10
  30. 2021Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategiescited by 9
  31. 2022Can Foundation Models Help Us Achieve Perfect Secrecy?cited by 9
  32. 2024DrJAX: Scalable and Differentiable MapReduce Primitives in JAXcited by 9
  33. 2022FLAME: Federated Learning Across Multi-device Environmentscited by 8
  34. 2022Auxo: Efficient Federated Learning via Scalable Client Clusteringcited by 8
  35. 2023VFLAIR: A Research Library and Benchmark for Vertical Federated Learningcited by 8
  36. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  37. 2024Promoting Data and Model Privacy in Federated Learning through Quantized LoRAcited by 8
  38. 2021FedTune: Automatic Tuning of Federated Learning Hyper-Parameters from System Perspectivecited by 7

50 most cited of 135 All 135 (JSON)