mlsanityData to 7 Oct 2026

A Comprehensive Empirical Study of Bugs in Open-Source Federated Learning Frameworks

arXiv 2308.05014 ↗First posted 9 Aug 2023cs.SE cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a distributed machine learning (ML) paradigm, allowing multiple clients to collaboratively train shared machine learning (ML) models without exposing clients' data privacy. It has gained substantial popularity in recent years, especially since the enforcement of data protection laws and regulations in many countries. To foster the application of FL, a variety of FL frameworks have been proposed, allowing non-experts to easily train ML models. As a result, understanding bugs in FL frameworks is critical for facilitating the development of better FL frameworks and potentially encouraging the development of bug detection, localization and repair tools. Thus, we conduct the first empirical study to comprehensively collect, taxonomize, and characterize bugs in FL frameworks. Specifically, we manually collect and classify 1,119 bugs from all the 676 closed issues and 514 merged pull requests in 17 popular and representative open-source FL frameworks on GitHub. We propose a classification of those bugs into 12 bug symptoms, 12 root causes, and 18 fix patterns. We also study their correlations and distributions on 23 functionalities. We identify nine major findings from our study, discuss their implications and future research directions based on our findings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 99 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

29 of 99 references.

References · 29

  1. 2019Towards Federated Learning at Scale: System Designcited by 826
  2. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 479
  3. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  4. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  5. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  6. 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 193
  7. 2018Mitigating Sybils in Federated Learning Poisoningcited by 158
  8. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  9. 2018A generic framework for privacy preserving deep learningcited by 103
  10. 2022NVIDIA FLARE: Federated Learning from Simulation to Real-Worldcited by 78
  11. 2022Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learningcited by 76
  12. 2019A Comprehensive Study on Deep Learning Bug Characteristicscited by 65
17 more
  1. 2021OpenFL: An open-source framework for Federated Learningcited by 58
  2. 2020IBM Federated Learning: an Enterprise Framework White Paper V0.1cited by 57
  3. 2021FedLab: A Flexible Federated Learning Frameworkcited by 54
  4. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  5. 2021FedJAX: Federated learning simulation with JAXcited by 32
  6. 2022FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulationscited by 27
  7. 2022Toward Understanding Deep Learning Framework Bugscited by 26
  8. 2018FixMiner: Mining Relevant Fix Patterns for Automated Program Repaircited by 22
  9. 2022APPFL: Open-Source Software Framework for Privacy-Preserving Federated Learningcited by 20
  10. 2021Scalable federated machine learning with FEDncited by 16
  11. 2019Substra: a framework for privacy-preserving, traceable and collaborative Machine Learningcited by 14
  12. 2020GFL: A Decentralized Federated Learning Framework Based On Blockchaincited by 12
  13. 2021A Survey on Federated Learning and its Applications for Accelerating Industrial Internet of Thingscited by 11
  14. 2021A Coupled Design of Exploiting Record Similarity for Practical Vertical Federated Learningcited by 10
  15. 2020Detecting and Understanding Real-World Differential Performance Bugs in Machine Learning Librariescited by 7
  16. 2023Flame: Simplifying Topology Extension in Federated Learningcited by 7
  17. 2022Towards Understanding the Faults of JavaScript-Based Deep Learning Systemscited by 6

Built on by

Not cited yet.