mlsanityData to 14 Oct 2026

The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems

arXiv 2006.07856 ↗First posted 14 Jun 2020cs.LG stat.MLcited by 31

Not on the map. Why?

Abstract (arXiv)

This paper presents and characterizes an Open Application Repository for Federated Learning (OARF), a benchmark suite for federated machine learning systems. Previously available benchmarks for federated learning have focused mainly on synthetic datasets and use a limited number of applications. OARF mimics more realistic application scenarios with publicly available data sets as different data silos in image, text and structured data. Our characterization shows that the benchmark suite is diverse in data size, distribution, feature distribution and learning task complexity. The extensive evaluations with reference implementations show the future research opportunities for important aspects of federated learning systems. We have developed reference implementations, and evaluated the important aspects of federated learning, including model accuracy, communication cost, throughput and convergence time. Through these evaluations, we discovered some interesting findings such as federated learning can effectively increase end-to-end throughput.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 10; 2024: 4; 2025: 4; 2026: 2.

31 papers.

Sort:

  1. 2021Model-Contrastive Federated Learningcited by 766
  2. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  3. 2021Towards Personalized Federated Learningcited by 464
  4. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  5. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  6. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  7. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  8. 2022FLAIR: Federated Learning Annotated Image Repositorycited by 22
  9. 2023PFLlib: A Beginner-Friendly and Comprehensive Personalized Federated Learning Library and Benchmarkcited by 17
  10. 2023A Review of the Role of Causality in Developing Trustworthy AI Systemscited by 14
  11. 2023FedBEVT: Federated Learning Bird's Eye View Perception Transformer in Road Traffic Systemscited by 14
  12. 2021Towards Unsupervised Domain Adaptation for Deep Face Recognition under Privacy Constraints via Federated Learningcited by 10
19 more
  1. 2021Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategiescited by 9
  2. 2022Optimizing Performance of Federated Person Re-identification: Benchmarking and Analysiscited by 9
  3. 2023Federated Learning for Clinical Structured Data: A Benchmark Comparison of Engineering and Statistical Approachescited by 8
  4. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  5. 2024COALA: A Practical and Vision-Centric Federated Learning Platformcited by 7
  6. 2025HtFLlib: A Comprehensive Heterogeneous Federated Learning Library and Benchmarkcited by 7
  7. 2023VertiBench: Advancing Feature Distribution Diversity in Vertical Federated Learning Benchmarkscited by 6
  8. 2023Federated Learning for Predictive Maintenance and Quality Inspection in Industrial Applicationscited by 4
  9. 2022UniFed: All-In-One Federated Learning Platform to Unify Open-Source Frameworkscited by 3
  10. 2022Federated clustering with GAN-based data synthesiscited by 3
  11. 2023Benchmarking FedAvg and FedCurv for Image Classification Taskscited by 3
  12. 2024Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Studycited by 2
  13. 2025Distributionally Robust Federated Learning with Client Drift Minimizationcited by 1
  14. 2023MetisFL: An Embarrassingly Parallelized Controller for Scalable & Efficient Federated Learning Workflowscited by 0
  15. 2024Where is the Testbed for my Federated Learning Research?cited by 0
  16. 2025WikiDBGraph: A Data Management Benchmark Suite for Collaborative Learning over Database Siloscited by 0
  17. 2025A Robust Federated Learning Approach for Combating Attacks Against IoT Systems Under non-IID Challengescited by 0
  18. 2026FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learningcited by 0
  19. 2026Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deploymentcited by 0