mlsanityData to 7 Oct 2026

Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing

arXiv 2106.04502 ↗First posted 8 Jun 2021cs.LG cs.AI cs.DCcited by 58

Not on the map. Why?

Abstract (arXiv)

Tuning hyperparameters is a crucial but arduous part of the machine learning pipeline. Hyperparameter optimization is even more challenging in federated learning, where models are learned over a distributed network of heterogeneous devices; here, the need to keep data on device and perform local training makes it difficult to efficiently train and evaluate configurations. In this work, we investigate the problem of federated hyperparameter tuning. We first identify key challenges and show how standard approaches may be adapted to form baselines for the federated setting. Then, by making a novel connection to the neural architecture search technique of weight-sharing, we introduce a new method, FedEx, to accelerate federated hyperparameter tuning that is applicable to widely-used federated optimization methods such as FedAvg and recent variants. Theoretically, we show that a FedEx variant correctly tunes the on-device learning rate in the setting of online convex optimization across devices. Empirically, we show that FedEx can outperform natural baselines for federated hyperparameter tuning by several percentage points on the Shakespeare, FEMNIST, and CIFAR-10 benchmarks, obtaining higher accuracy using the same training budget.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 37 of its 64 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

37 of 64 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  10. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  11. 2018LEAF: A Benchmark for Federated Settingscited by 780
  12. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
25 more
  1. 2017Federated Multi-Task Learningcited by 669
  2. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  3. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  4. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  5. 2019Fair Resource Allocation in Federated Learningcited by 389
  6. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  7. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  8. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  9. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  10. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  11. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  12. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  13. 2020Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithmscited by 78
  14. 2020Federated Bayesian Optimization via Thompson Samplingcited by 77
  15. 2019Meta-Learning of Neural Architectures for Few-Shot Learningcited by 73
  16. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  17. 2015A PAC Approach to Application-Specific Algorithm Selectioncited by 57
  18. 2019Semi-Cyclic Stochastic Gradient Descentcited by 40
  19. 2019Differentially Private Meta-Learningcited by 40
  20. 2020Geometry-Aware Gradient Algorithms for Neural Architecture Searchcited by 40
  21. 2019Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameterscited by 16
  22. 2020Federated Neural Architecture Searchcited by 10
  23. 2020Guarantees for Tuning the Step Size using a Learning-to-Learn Approachcited by 10
  24. 2018Learning Rate Adaptation for Federated and Differentially Private Learningcited by 7
  25. 2020Direct Federated Neural Architecture Searchcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 19; 2023: 16; 2024: 9; 2025: 3; 2026: 5.

58 papers.

Sort:

  1. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  2. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  3. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  4. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  5. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  6. 2022FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph Learningcited by 43
  7. 2021Differentially Private Federated Bayesian Optimization with Distributed Explorationcited by 32
  8. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  9. 2022Learning Predictions for Algorithms with Predictionscited by 25
  10. 2023Adaptive Test-Time Personalization for Federated Learningcited by 23
  11. 2021GIFAIR-FL: A Framework for Group and Individual Fairness in Federated Learningcited by 21
  12. 2023Achieving Linear Speedup in Non-IID Federated Bilevel Learningcited by 18
38 more
  1. 2022Federated Neural Banditscited by 17
  2. 2022Auto-FedRL: Federated Hyperparameter Optimization for Multi-institutional Medical Image Segmentationcited by 16
  3. 2022Self-Aware Personalized Federated Learningcited by 16
  4. 2021Learning-to-learn non-convex piecewise-Lipschitz functionscited by 15
  5. 2021FLoRA: Single-shot Hyper-parameter Optimization for Federated Learningcited by 14
  6. 2024Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptationcited by 12
  7. 2023Making Batch Normalization Great in Federated Deep Learningcited by 8
  8. 2023Hyperparameter Optimization through Neural Network Partitioningcited by 8
  9. 2023Meta-Learning Adversarial Bandit Algorithmscited by 8
  10. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  11. 2021FedTune: Automatic Tuning of Federated Learning Hyper-Parameters from System Perspectivecited by 7
  12. 2022FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimizationcited by 6
  13. 2022On Noisy Evaluation in Federated Hyperparameter Tuningcited by 6
  14. 2022Meta-Learning Adversarial Banditscited by 5
  15. 2022Federated X-Armed Banditcited by 5
  16. 2022Supernet Training for Federated Image Classification under System Heterogeneitycited by 4
  17. 2023FedLAP-DP: Federated Learning by Sharing Differentially Private Loss Approximationscited by 4
  18. 2023Combating Exacerbated Heterogeneity for Robust Models in Federated Learningcited by 4
  19. 2024Federated Learning Client Pruning for Noisy Labelscited by 4
  20. 2022Single-shot Hyper-parameter Optimization for Federated Learning: A General Algorithm & Analysiscited by 3
  21. 2022FEATHERS: Federated Architecture and Hyperparameter Searchcited by 3
  22. 2022FedGPO: Heterogeneity-Aware Global Parameter Optimization for Efficient Federated Learningcited by 3
  23. 2023A review of federated learning in renewable energy applications: Potential, challenges, and future directionscited by 3
  24. 2024Wind turbine condition monitoring based on intra- and inter-farm federated learningcited by 3
  25. 2022Federated Hypergradient Descentcited by 2
  26. 2023Federated Automatic Differentiationcited by 2
  27. 2023FedAVO: Improving Communication Efficiency in Federated Learning with African Vultures Optimizercited by 2
  28. 2023FedECA: Federated External Control Arms for Causal Inference with Time-To-Event Data in Distributed Settingscited by 2
  29. 2022Federated Learning Hyper-Parameter Tuning from a System Perspectivecited by 1
  30. 2023HPN: Personalized Federated Hyperparameter Optimizationcited by 1
  31. 2023FedPop: Federated Population-based Hyperparameter Tuningcited by 1
  32. 2024SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learningcited by 1
  33. 2024Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanismcited by 1
  34. 2025LNUCB-TA: Linear-nonlinear Hybrid Bandit Learning with Temporal Attentioncited by 1
  35. 2021Boosting Resource-Constrained Federated Learning Systems with Guessed Updatescited by 0
  36. 2023Hybrid Algorithm Selection and Hyperparameter Tuning on Distributed Machine Learning Resources: A Hierarchical Agent-based Approachcited by 0
  37. 2023FedL2P: Federated Learning to Personalizecited by 0
  38. 2024Harnessing Increased Client Participation with Cohort-Parallel Federated Learningcited by 0

50 most cited of 58 All 58 (JSON)