mlsanityData to 14 Oct 2026

FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization

arXiv 2206.03966 ↗First posted 8 Jun 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Hyperparameter optimization (HPO) is crucial for machine learning algorithms to achieve satisfactory performance, whose progress has been boosted by related benchmarks. Nonetheless, existing efforts in benchmarking all focus on HPO for traditional centralized learning while ignoring federated learning (FL), a promising paradigm for collaboratively learning models from dispersed data. In this paper, we first identify some uniqueness of HPO for FL algorithms from various aspects. Due to this uniqueness, existing HPO benchmarks no longer satisfy the need to compare HPO methods in the FL setting. To facilitate the research of HPO in the FL setting, we propose and implement a benchmark suite FedHPO-B that incorporates comprehensive FL tasks, enables efficient function evaluations, and eases continuing extensions. We also conduct extensive experiments based on FedHPO-B to benchmark a few HPO methods. We open-source FedHPO-B at https://github.com/alibaba/FederatedScope/tree/master/benchmark/FedHPOB.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 8 Jun 2022.

Builds on

32 of 50 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  7. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  8. 2018LEAF: A Benchmark for Federated Settingscited by 792
  9. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  10. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  11. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  12. 2014OpenML: networked science in machine learningcited by 526
20 more
  1. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  2. 2021A Field Guide to Federated Optimizationcited by 315
  3. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 230
  4. 2017OpenML Benchmarking Suitescited by 178
  5. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  6. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 132
  7. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  8. 2014Freeze-Thaw Bayesian Optimizationcited by 90
  9. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 90
  10. 2019An Open Source AutoML Benchmarkcited by 85
  11. 2020Federated Bayesian Optimization via Thompson Samplingcited by 77
  12. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  13. 2021YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimizationcited by 62
  14. 2021Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharingcited by 58
  15. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  16. 2022FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph Learningcited by 43
  17. 2019Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameterscited by 16
  18. 2021FLoRA: Single-shot Hyper-parameter Optimization for Federated Learningcited by 14
  19. 2020Differential Evolution for Neural Architecture Searchcited by 10
  20. 2021FedTune: Automatic Tuning of Federated Learning Hyper-Parameters from System Perspectivecited by 7

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 3.

6 papers.

  1. 2023FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descentcited by 8
  2. 2022On Noisy Evaluation in Federated Hyperparameter Tuningcited by 7
  3. 2023FLEdge: Benchmarking Federated Machine Learning Applications in Edge Computing Systemscited by 3
  4. 2022Federated Hypergradient Descentcited by 2
  5. 2022Federated Learning Hyper-Parameter Tuning from a System Perspectivecited by 1
  6. 2023HPN: Personalized Federated Hyperparameter Optimizationcited by 1