# Evaluation Framework For Large-scale Federated Learning

arXiv: https://arxiv.org/abs/2003.01575 · HTML: https://mlsanity.com/paper/2003.01575/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2020.

## Built on by (most cited first, 4 of 4)

- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Benchmarking FedAvg and FedCurv for Image Classification Tasks](https://mlsanity.com/paper/2303.17942/) (2023)
- [Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learning](https://mlsanity.com/paper/2506.18789/) (2025)
- [FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learning](https://mlsanity.com/paper/2605.07962/) (2026)

## Abstract (arXiv)

Federated learning is proposed as a machine learning setting to enable distributed edge devices, such as mobile phones, to collaboratively learn a shared prediction model while keeping all the training data on device, which can not only take full advantage of data distributed across millions of nodes to train a good model but also protect data privacy. However, learning in scenario above poses new challenges. In fact, data across a massive number of unreliable devices is likely to be non-IID (identically and independently distributed), which may make the performance of models trained by federated learning unstable. In this paper, we introduce a framework designed for large-scale federated learning which consists of approaches to generating dataset and modular evaluation framework. Firstly, we construct a suite of open-source non-IID datasets by providing three respects including covariate shift, prior probability shift, and concept shift, which are grounded in real-world assumptions. In addition, we design several rigorous evaluation metrics including the number of network nodes, the size of datasets, the number of communication rounds and communication resources etc. Finally, we present an open-source benchmark for large-scale federated learning research.