mlsanityData to 7 Oct 2026

On Noisy Evaluation in Federated Hyperparameter Tuning

arXiv 2212.08930 ↗First posted 17 Dec 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Hyperparameter tuning is critical to the success of federated learning applications. Unfortunately, appropriately selecting hyperparameters is challenging in federated networks. Issues of scale, privacy, and heterogeneity introduce noise in the tuning process and make it difficult to evaluate the performance of various hyperparameters. In this work, we perform the first systematic study on the effect of noisy evaluation in federated hyperparameter tuning. We first identify and rigorously explore key sources of noise, including client subsampling, data and systems heterogeneity, and data privacy. Surprisingly, our results indicate that even small amounts of noise can significantly impact tuning methods-reducing the performance of state-of-the-art approaches to that of naive baselines. To address noisy evaluation in such scenarios, we propose a simple and effective approach that leverages public proxy data to boost the evaluation signal. Our work establishes general challenges, baselines, and best practices for future work in federated hyperparameter tuning.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 41 of its 64 references that are in the map. First posted on arXiv 17 Dec 2022.

Builds on

41 of 64 references.

References · 41

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  6. 2020Adaptive Federated Optimizationcited by 1,036
  7. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  8. 2019Towards Federated Learning at Scale: System Designcited by 826
  9. 2018LEAF: A Benchmark for Federated Settingscited by 780
  10. 2018A Tutorial on Bayesian Optimizationcited by 734
  11. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  12. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
29 more
  1. 2019Agnostic Federated Learningcited by 451
  2. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  3. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  4. 2019Fair Resource Allocation in Federated Learningcited by 389
  5. 2017Population Based Training of Neural Networkscited by 351
  6. 2021A Field Guide to Federated Optimizationcited by 312
  7. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  8. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 131
  9. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  10. 2017Constrained Bayesian Optimization with Noisy Experimentscited by 103
  11. 2019Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuningcited by 91
  12. 2016Multi-Information Source Optimizationcited by 83
  13. 2020Federated Bayesian Optimization via Thompson Samplingcited by 77
  14. 2018Private Selection from Private Candidatescited by 75
  15. 2021On Large-Cohort Training for Federated Learningcited by 73
  16. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  17. 2021Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharingcited by 58
  18. 2021Differentially Private Federated Bayesian Optimization with Distributed Explorationcited by 32
  19. 2022Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneitycited by 17
  20. 2019Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameterscited by 16
  21. 2021Oneshot Differentially Private Top-k Selectioncited by 16
  22. 2021FLoRA: Single-shot Hyper-parameter Optimization for Federated Learningcited by 14
  23. 2015Differentially Private Bayesian Optimizationcited by 12
  24. 2021Evaluation of Hyperparameter-Optimization Approaches in an Industrial Federated Learning Systemcited by 8
  25. 2018Learning Rate Adaptation for Federated and Differentially Private Learningcited by 7
  26. 2021FedTune: Automatic Tuning of Federated Learning Hyper-Parameters from System Perspectivecited by 7
  27. 2022FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimizationcited by 6
  28. 2020Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learningcited by 5
  29. 2022Dap-FL: Federated Learning flourishes by adaptive tuning and secure aggregationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 2; 2026: 1.

6 papers.

  1. 2023FLINT: A Platform for Federated Learning Integrationcited by 17
  2. 2023Leveraging Function Space Aggregation for Federated Learning at Scalecited by 4
  3. 2023A Bayesian approach for prompt optimization in pre-trained language modelscited by 4
  4. 2025Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practicecited by 3
  5. 2025crowd-hpo: Realistic Hyperparameter Optimization and Benchmarking for Learning from Crowds with Noisy Labelscited by 0
  6. 2026Generalizable Hyperparameter Optimization for Federated Learning on Non-IID Cancer Imagescited by 0