# FedEval: A Holistic Evaluation Framework for Federated Learning

arXiv: https://arxiv.org/abs/2011.09655 · HTML: https://mlsanity.com/paper/2011.09655/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2020.

## Built on by (most cited first, 9 of 9)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [PFLlib: A Beginner-Friendly and Comprehensive Personalized Federated Learning Library and Benchmark](https://mlsanity.com/paper/2312.04992/) (2023)
- [A Survey for Federated Learning Evaluations: Goals and Measures](https://mlsanity.com/paper/2308.11841/) (2023)
- [pfl-research: simulation framework for accelerating research in Private Federated Learning](https://mlsanity.com/paper/2404.06430/) (2024)
- [Federated Short-Term Load Forecasting with Personalization Layers for Heterogeneous Clients](https://mlsanity.com/paper/2309.13194/) (2023)
- [Holistic Evaluation Metrics: Use Case Sensitive Evaluation Metrics for Federated Learning](https://mlsanity.com/paper/2405.02360/) (2024)
- [PracMHBench: Re-evaluating Model-Heterogeneous Federated Learning Based on Practical Edge Device Constraints](https://mlsanity.com/paper/2509.08750/) (2025)
- [FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learning](https://mlsanity.com/paper/2605.07962/) (2026)
- [Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment](https://mlsanity.com/paper/2608.25114/) (2026)

## Abstract (arXiv)

Federated Learning (FL) has been widely accepted as the solution for privacy-preserving machine learning without collecting raw data. While new technologies proposed in the past few years do evolve the FL area, unfortunately, the evaluation results presented in these works fall short in integrity and are hardly comparable because of the inconsistent evaluation metrics and experimental settings. In this paper, we propose a holistic evaluation framework for FL called FedEval, and present a benchmarking study on seven state-of-the-art FL algorithms. Specifically, we first introduce the core evaluation taxonomy model, called FedEval-Core, which covers four essential evaluation aspects for FL: Privacy, Robustness, Effectiveness, and Efficiency, with various well-defined metrics and experimental settings. Based on the FedEval-Core, we further develop an FL evaluation platform with standardized evaluation settings and easy-to-use interfaces. We then provide an in-depth benchmarking study between the seven well-known FL algorithms, including FedSGD, FedAvg, FedProx, FedOpt, FedSTC, SecAgg, and HEAgg. We comprehensively analyze the advantages and disadvantages of these algorithms and further identify the suitable practical scenarios for different algorithms, which is rarely done by prior work. Lastly, we excavate a set of take-away insights and future research directions, which are very helpful for researchers in the FL area.