# The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems

arXiv: https://arxiv.org/abs/2006.07856 · HTML: https://mlsanity.com/paper/2006.07856/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2020.

## Built on by (most cited first, 30 of 31)

- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [FedScale: Benchmarking Model and System Performance of Federated Learning at Scale](https://mlsanity.com/paper/2105.11367/) (2021)
- [Motley: Benchmarking Heterogeneity and Personalization in Federated Learning](https://mlsanity.com/paper/2206.09262/) (2022)
- [Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning](https://mlsanity.com/paper/2307.09619/) (2023)
- [FLAIR: Federated Learning Annotated Image Repository](https://mlsanity.com/paper/2207.08869/) (2022)
- [PFLlib: A Beginner-Friendly and Comprehensive Personalized Federated Learning Library and Benchmark](https://mlsanity.com/paper/2312.04992/) (2023)
- [A Review of the Role of Causality in Developing Trustworthy AI Systems](https://mlsanity.com/paper/2302.06975/) (2023)
- [FedBEVT: Federated Learning Bird's Eye View Perception Transformer in Road Traffic Systems](https://mlsanity.com/paper/2304.01534/) (2023)
- [Towards Unsupervised Domain Adaptation for Deep Face Recognition under Privacy Constraints via Federated Learning](https://mlsanity.com/paper/2105.07606/) (2021)
- [Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies](https://mlsanity.com/paper/2109.03999/) (2021)
- [Optimizing Performance of Federated Person Re-identification: Benchmarking and Analysis](https://mlsanity.com/paper/2205.12144/) (2022)
- [Federated Learning for Clinical Structured Data: A Benchmark Comparison of Engineering and Statistical Approaches](https://mlsanity.com/paper/2311.03417/) (2023)
- [pfl-research: simulation framework for accelerating research in Private Federated Learning](https://mlsanity.com/paper/2404.06430/) (2024)
- [COALA: A Practical and Vision-Centric Federated Learning Platform](https://mlsanity.com/paper/2407.16560/) (2024)
- [HtFLlib: A Comprehensive Heterogeneous Federated Learning Library and Benchmark](https://mlsanity.com/paper/2506.03954/) (2025)
- [VertiBench: Advancing Feature Distribution Diversity in Vertical Federated Learning Benchmarks](https://mlsanity.com/paper/2307.02040/) (2023)
- [Federated Learning for Predictive Maintenance and Quality Inspection in Industrial Applications](https://mlsanity.com/paper/2304.11101/) (2023)
- [UniFed: All-In-One Federated Learning Platform to Unify Open-Source Frameworks](https://mlsanity.com/paper/2207.10308/) (2022)
- [Federated clustering with GAN-based data synthesis](https://mlsanity.com/paper/2210.16524/) (2022)
- [Benchmarking FedAvg and FedCurv for Image Classification Tasks](https://mlsanity.com/paper/2303.17942/) (2023)
- [Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2402.12876/) (2024)
- [Distributionally Robust Federated Learning with Client Drift Minimization](https://mlsanity.com/paper/2505.15371/) (2025)
- [MetisFL: An Embarrassingly Parallelized Controller for Scalable & Efficient Federated Learning Workflows](https://mlsanity.com/paper/2311.00334/) (2023)
- [Where is the Testbed for my Federated Learning Research?](https://mlsanity.com/paper/2407.14154/) (2024)
- [WikiDBGraph: A Data Management Benchmark Suite for Collaborative Learning over Database Silos](https://mlsanity.com/paper/2505.16635/) (2025)
- [A Robust Federated Learning Approach for Combating Attacks Against IoT Systems Under non-IID Challenges](https://mlsanity.com/paper/2511.16822/) (2025)
- [FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learning](https://mlsanity.com/paper/2605.07962/) (2026)

## Abstract (arXiv)

This paper presents and characterizes an Open Application Repository for Federated Learning (OARF), a benchmark suite for federated machine learning systems. Previously available benchmarks for federated learning have focused mainly on synthetic datasets and use a limited number of applications. OARF mimics more realistic application scenarios with publicly available data sets as different data silos in image, text and structured data. Our characterization shows that the benchmark suite is diverse in data size, distribution, feature distribution and learning task complexity. The extensive evaluations with reference implementations show the future research opportunities for important aspects of federated learning systems. We have developed reference implementations, and evaluated the important aspects of federated learning, including model accuracy, communication cost, throughput and convergence time. Through these evaluations, we discovered some interesting findings such as federated learning can effectively increase end-to-end throughput.