# A Comprehensive Empirical Study of Bugs in Open-Source Federated Learning Frameworks

arXiv: https://arxiv.org/abs/2308.05014 · HTML: https://mlsanity.com/paper/2308.05014/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 99 references that are in the map.
- First posted on arXiv 9 Aug 2023.

## Builds on (most cited first, 29 of 29)

- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Local Model Poisoning Attacks to Byzantine-Robust Federated Learning](https://mlsanity.com/paper/1911.11815/) (2019)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [FedML: A Research Library and Benchmark for Federated Machine Learning](https://mlsanity.com/paper/2007.13518/) (2020)
- [Federated Optimization:Distributed Optimization Beyond the Datacenter](https://mlsanity.com/paper/1511.03575/) (2015)
- [Mitigating Sybils in Federated Learning Poisoning](https://mlsanity.com/paper/1808.04866/) (2018)
- [FedScale: Benchmarking Model and System Performance of Federated Learning at Scale](https://mlsanity.com/paper/2105.11367/) (2021)
- [A generic framework for privacy preserving deep learning](https://mlsanity.com/paper/1811.04017/) (2018)
- [NVIDIA FLARE: Federated Learning from Simulation to Real-World](https://mlsanity.com/paper/2210.13291/) (2022)
- [Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning](https://mlsanity.com/paper/2207.08281/) (2022)
- [A Comprehensive Study on Deep Learning Bug Characteristics](https://mlsanity.com/paper/1906.01388/) (2019)
- [OpenFL: An open-source framework for Federated Learning](https://mlsanity.com/paper/2105.06413/) (2021)
- [IBM Federated Learning: an Enterprise Framework White Paper V0.1](https://mlsanity.com/paper/2007.10987/) (2020)
- [FedLab: A Flexible Federated Learning Framework](https://mlsanity.com/paper/2107.11621/) (2021)
- [FederatedScope: A Flexible Federated Learning Platform for Heterogeneity](https://mlsanity.com/paper/2204.05011/) (2022)
- [FedJAX: Federated learning simulation with JAX](https://mlsanity.com/paper/2108.02117/) (2021)
- [FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulations](https://mlsanity.com/paper/2203.13789/) (2022)
- [Toward Understanding Deep Learning Framework Bugs](https://mlsanity.com/paper/2203.04026/) (2022)
- [FixMiner: Mining Relevant Fix Patterns for Automated Program Repair](https://mlsanity.com/paper/1810.01791/) (2018)
- [APPFL: Open-Source Software Framework for Privacy-Preserving Federated Learning](https://mlsanity.com/paper/2202.03672/) (2022)
- [Scalable federated machine learning with FEDn](https://mlsanity.com/paper/2103.00148/) (2021)
- [Substra: a framework for privacy-preserving, traceable and collaborative Machine Learning](https://mlsanity.com/paper/1910.11567/) (2019)
- [GFL: A Decentralized Federated Learning Framework Based On Blockchain](https://mlsanity.com/paper/2010.10996/) (2020)
- [A Survey on Federated Learning and its Applications for Accelerating Industrial Internet of Things](https://mlsanity.com/paper/2104.10501/) (2021)
- [A Coupled Design of Exploiting Record Similarity for Practical Vertical Federated Learning](https://mlsanity.com/paper/2106.06312/) (2021)
- [Detecting and Understanding Real-World Differential Performance Bugs in Machine Learning Libraries](https://mlsanity.com/paper/2006.01991/) (2020)
- [Flame: Simplifying Topology Extension in Federated Learning](https://mlsanity.com/paper/2305.05118/) (2023)
- [Towards Understanding the Faults of JavaScript-Based Deep Learning Systems](https://mlsanity.com/paper/2209.04791/) (2022)

## Abstract (arXiv)

Federated learning (FL) is a distributed machine learning (ML) paradigm, allowing multiple clients to collaboratively train shared machine learning (ML) models without exposing clients' data privacy. It has gained substantial popularity in recent years, especially since the enforcement of data protection laws and regulations in many countries. To foster the application of FL, a variety of FL frameworks have been proposed, allowing non-experts to easily train ML models. As a result, understanding bugs in FL frameworks is critical for facilitating the development of better FL frameworks and potentially encouraging the development of bug detection, localization and repair tools. Thus, we conduct the first empirical study to comprehensively collect, taxonomize, and characterize bugs in FL frameworks. Specifically, we manually collect and classify 1,119 bugs from all the 676 closed issues and 514 merged pull requests in 17 popular and representative open-source FL frameworks on GitHub. We propose a classification of those bugs into 12 bug symptoms, 12 root causes, and 18 fix patterns. We also study their correlations and distributions on 23 functionalities. We identify nine major findings from our study, discuss their implications and future research directions based on our findings.