mlsanityData to 14 Oct 2026

Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment

arXiv 2608.25114 ↗First posted 25 Aug 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) has emerged as a key approach for training models across decentralized data, yet benchmarking in FL remains difficult to reproduce, compare, and extend. Existing evaluations are often tied to custom infrastructure, released as incomplete research code, and conducted primarily in simulation, which limits portability and practical relevance. We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications. We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation workflows. We instantiate this approach with a multi-domain benchmark suite spanning cross-silo and cross-device settings, and including tasks in medical imaging, financial tabular learning, legal instruction tuning, phishing URL detection, and audio tagging. We further demonstrate that the same benchmarking application can run across both simulation and deployment runtimes without changing the application code, enabling unified evaluation across varying learning environments. Beyond model quality, our benchmark design supports system-aware reporting, including runtime and communication metrics. This work advances benchmarking in FL settings from ad hoc code artifacts towards portable, executable, and reusable benchmark applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 48 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

38 of 48 references.

References · 38

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  3. 2018Federated Learning with Non-IID Datacited by 1,108
  4. 2020Adaptive Federated Optimizationcited by 1,046
  5. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  6. 2018LEAF: A Benchmark for Federated Settingscited by 792
  7. 2021Model-Contrastive Federated Learningcited by 766
  8. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  9. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  10. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  11. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  12. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
26 more
  1. 2023FinGPT: Open-Source Financial Large Language Modelscited by 253
  2. 2024Capabilities of Gemini Models in Medicinecited by 237
  3. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  4. 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 125
  5. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 120
  6. 2021MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluationcited by 88
  7. 2021Federated Learning: Opportunities and Challengescited by 84
  8. 2022NVIDIA FLARE: Federated Learning from Simulation to Real-Worldcited by 80
  9. 2021FedLab: A Flexible Federated Learning Frameworkcited by 55
  10. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  11. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  12. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  13. 2016Automatic tagging using deep convolutional neural networkscited by 40
  14. 2021Emerging Trends in Federated Learning: From Model Fusion to Federated X Learningcited by 35
  15. 2022Faster Adaptive Federated Learningcited by 35
  16. 2018URLNet: Learning a URL Representation with Deep Learning for Malicious URL Detectioncited by 34
  17. 2020The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systemscited by 31
  18. 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 30
  19. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
  20. 2022FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulationscited by 27
  21. 2021EasyFL: A Low-code Federated Learning Platform For Dummiescited by 20
  22. 2022APPFL: Open-Source Software Framework for Privacy-Preserving Federated Learningcited by 20
  23. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  24. 2020FedEval: A Holistic Evaluation Framework for Federated Learningcited by 9
  25. 2023FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Thingscited by 8
  26. 2022UniFed: All-In-One Federated Learning Platform to Unify Open-Source Frameworkscited by 3

Built on by

Not cited yet.