mlsanityData to 14 Oct 2026

Where is the Testbed for my Federated Learning Research?

arXiv 2407.14154 ↗First posted 19 Jul 2024cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Progressing beyond centralized AI is of paramount importance, yet, distributed AI solutions, in particular various federated learning (FL) algorithms, are often not comprehensively assessed, which prevents the research community from identifying the most promising approaches and practitioners from being convinced that a certain solution is deployment-ready. The largest hurdle towards FL algorithm evaluation is the difficulty of conducting real-world experiments over a variety of FL client devices and different platforms, with different datasets and data distribution, all while assessing various dimensions of algorithm performance, such as inference accuracy, energy consumption, and time to convergence, to name a few. In this paper, we present CoLExT, a real-world testbed for FL research. CoLExT is designed to streamline experimentation with custom FL algorithms in a rich testbed configuration space, with a large number of heterogeneous edge devices, ranging from single-board computers to smartphones, and provides real-time collection and visualization of a variety of metrics through automatic instrumentation. According to our evaluation, porting FL algorithms to CoLExT requires minimal involvement from the developer, and the instrumentation introduces minimal resource usage overhead. Furthermore, through an initial investigation involving popular FL algorithms running on CoLExT, we reveal previously unknown trade-offs, inefficiencies, and programming bugs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 87 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

50 of 87 references.

References · 50

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2020Adaptive Federated Optimizationcited by 1,046
  5. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  6. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  7. 2019Towards Federated Learning at Scale: System Designcited by 832
  8. 2018LEAF: A Benchmark for Federated Settingscited by 792
  9. 2021Model-Contrastive Federated Learningcited by 766
  10. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  11. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  12. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
38 more
  1. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
  2. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  3. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  4. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
  5. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  6. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  7. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  8. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  9. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  10. 2020Federated Visual Classification with Real-World Data Distributioncited by 128
  11. 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 125
  12. 2022NVIDIA FLARE: Federated Learning from Simulation to Real-Worldcited by 80
  13. 2021OpenFL: An open-source framework for Federated Learningcited by 58
  14. 2020IBM Federated Learning: an Enterprise Framework White Paper V0.1cited by 57
  15. 2021FedLab: A Flexible Federated Learning Frameworkcited by 55
  16. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  17. 2022FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clientscited by 46
  18. 2021FedJAX: Federated learning simulation with JAXcited by 32
  19. 2020The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systemscited by 31
  20. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  21. 2022FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulationscited by 27
  22. 2021FedAdapt: Adaptive Offloading for IoT Devices in Federated Learningcited by 26
  23. 2021Resource-Efficient Federated Learningcited by 26
  24. 2020FLaaS: Federated Learning as a Servicecited by 22
  25. 2022FLAIR: Federated Learning Annotated Image Repositorycited by 22
  26. 2023FS-Real: Towards Real-World Cross-Device Federated Learningcited by 22
  27. 2021EasyFL: A Low-code Federated Learning Platform For Dummiescited by 20
  28. 2019Substra: a framework for privacy-preserving, traceable and collaborative Machine Learningcited by 14
  29. 2020PrivacyFL: A simulator for privacy-preserving and secure federated learningcited by 12
  30. 2023Asynchronous Federated Learning with Bidirectional Quantized Communications and Buffered Aggregationcited by 10
  31. 2024Not All Federated Learning Algorithms Are Created Equal: A Performance Evaluation Studycited by 10
  32. 2023Flame: Simplifying Topology Extension in Federated Learningcited by 7
  33. 2023FilFL: Client Filtering for Optimized Client Participation in Federated Learningcited by 5
  34. 2023An Empirical Study of Federated Learning on IoT-Edge Devices: Resource Allocation and Heterogeneitycited by 5
  35. 2022Protea: Client Profiling within Federated Systems using Flowercited by 4
  36. 2024Flashback: Understanding and Mitigating Forgetting in Federated Learningcited by 4
  37. 2023FLEdge: Benchmarking Federated Machine Learning Applications in Edge Computing Systemscited by 3
  38. 2023FedHC: A Scalable Federated Learning Framework for Heterogeneous and Resource-Constrained Clientscited by 1

Built on by

Not cited yet.