# Not All Federated Learning Algorithms Are Created Equal: A Performance Evaluation Study

arXiv: https://arxiv.org/abs/2403.17287 · HTML: https://mlsanity.com/paper/2403.17287/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Federated Optimization: Distributed Machine Learning for On-Device Intelligence](https://mlsanity.com/paper/1610.02527/) (2016)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Fair Resource Allocation in Federated Learning](https://mlsanity.com/paper/1905.10497/) (2019)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Federated Learning with Buffered Asynchronous Aggregation](https://mlsanity.com/paper/2106.06639/) (2021)
- [FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction](https://mlsanity.com/paper/2203.11751/) (2022)
- [FairFed: Enabling Group Fairness in Federated Learning](https://mlsanity.com/paper/2110.00857/) (2021)
- [Hierarchical Federated Learning Across Heterogeneous Cellular Networks](https://mlsanity.com/paper/1909.02362/) (2019)
- [On Large-Cohort Training for Federated Learning](https://mlsanity.com/paper/2106.07820/) (2021)
- [Mitigating Bias in Federated Learning](https://mlsanity.com/paper/2012.02447/) (2020)
- [FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clients](https://mlsanity.com/paper/2201.01601/) (2022)
- [Training Fair Models in Federated Learning without Data Privacy Infringement](https://mlsanity.com/paper/2109.05662/) (2021)
- [Resource-Efficient Federated Learning](https://mlsanity.com/paper/2111.01108/) (2021)
- [Mitigating Group Bias in Federated Learning: Beyond Local Fairness](https://mlsanity.com/paper/2305.09931/) (2023)
- [Flame: Simplifying Topology Extension in Federated Learning](https://mlsanity.com/paper/2305.05118/) (2023)

## Built on by (most cited first, 10 of 10)

- [DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models](https://mlsanity.com/paper/2509.09097/) (2025)
- [CLLoRA: An Approach to Measure the Effects of the Context Length for LLM Fine-Tuning](https://mlsanity.com/paper/2502.18910/) (2025)
- [GC-Fed: Gradient Centralized Federated Learning with Partial Client Participation](https://mlsanity.com/paper/2503.13180/) (2025)
- [SAFE: Self-Adjustment Federated Learning Framework for Remote Sensing Collaborative Perception](https://mlsanity.com/paper/2504.03700/) (2025)
- [Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning with Real-Time Feedback](https://mlsanity.com/paper/2605.07977/) (2026)
- [Where is the Testbed for my Federated Learning Research?](https://mlsanity.com/paper/2407.14154/) (2024)
- [FedVSR: Towards Model-Agnostic Federated Learning in Video Super-Resolution](https://mlsanity.com/paper/2503.13745/) (2025)
- [SimDeep: Federated 3D Indoor Localization via Similarity-Aware Aggregation](https://mlsanity.com/paper/2508.01515/) (2025)
- [How Edge of Stability Hinders SCAFFOLD in Federated Optimization](https://mlsanity.com/paper/2608.25873/) (2026)
- [Pushing the (Decision) Boundaries: Dynamically Calibrating Differentially Private Noise to Explainability in Federated Learning](https://mlsanity.com/paper/2609.03851/) (2026)

## Abstract (arXiv)

Federated Learning (FL) emerged as a practical approach to training a model from decentralized data. The proliferation of FL led to the development of numerous FL algorithms and mechanisms. Many prior efforts have given their primary focus on accuracy of those approaches, but there exists little understanding of other aspects such as computational overheads, performance and training stability, etc. To bridge this gap, we conduct extensive performance evaluation on several canonical FL algorithms (FedAvg, FedProx, FedYogi, FedAdam, SCAFFOLD, and FedDyn) by leveraging an open-source federated learning framework called Flame. Our comprehensive measurement study reveals that no single algorithm works best across different performance metrics. A few key observations are: (1) While some state-of-the-art algorithms achieve higher accuracy than others, they incur either higher computation overheads (FedDyn) or communication overheads (SCAFFOLD). (2) Recent algorithms present smaller standard deviation in accuracy across clients than FedAvg, indicating that the advanced algorithms' performances are stable. (3) However, algorithms such as FedDyn and SCAFFOLD are more prone to catastrophic failures without the support of additional techniques such as gradient clipping. We hope that our empirical study can help the community to build best practices in evaluating FL algorithms.