# FLoRA: Single-shot Hyper-parameter Optimization for Federated Learning

arXiv: https://arxiv.org/abs/2112.08524 · HTML: https://mlsanity.com/paper/2112.08524/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 15 Dec 2021.

## Builds on (most cited first, 30 of 31)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [A Hybrid Approach to Privacy-Preserving Federated Learning](https://mlsanity.com/paper/1812.03224/) (2018)
- [SCAFFOLD: Stochastic Controlled Averaging for Federated Learning](https://mlsanity.com/paper/1910.06378/) (2019)
- [cpSGD: Communication-efficient and differentially-private distributed SGD](https://mlsanity.com/paper/1805.10559/) (2018)
- [TiFL: A Tier-based Federated Learning System](https://mlsanity.com/paper/2001.09249/) (2020)
- [Federated Bayesian Optimization via Thompson Sampling](https://mlsanity.com/paper/2010.10154/) (2020)
- [Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing](https://mlsanity.com/paper/2106.04502/) (2021)
- [IBM Federated Learning: an Enterprise Framework White Paper V0.1](https://mlsanity.com/paper/2007.10987/) (2020)
- [Mitigating Bias in Federated Learning](https://mlsanity.com/paper/2012.02447/) (2020)
- [HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning](https://mlsanity.com/paper/1912.05897/) (2019)
- [Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning](https://mlsanity.com/paper/1909.12552/) (2019)
- [An ADMM Based Framework for AutoML Pipeline Configuration](https://mlsanity.com/paper/1905.00424/) (2019)
- [Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameters](https://mlsanity.com/paper/1912.13075/) (2019)
- [Selecting Near-Optimal Learners via Incremental Data Allocation](https://mlsanity.com/paper/1601.00024/) (2015)
- [Adaptive Histogram-Based Gradient Boosted Trees for Federated Learning](https://mlsanity.com/paper/2012.06670/) (2020)

## Built on by (most cited first, 14 of 14)

- [FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning](https://mlsanity.com/paper/2309.00363/) (2023)
- [Federated Neural Bandits](https://mlsanity.com/paper/2205.14309/) (2022)
- [Hyperparameter Optimization through Neural Network Partitioning](https://mlsanity.com/paper/2304.14766/) (2023)
- [FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descent](https://mlsanity.com/paper/2310.03156/) (2023)
- [On Noisy Evaluation in Federated Hyperparameter Tuning](https://mlsanity.com/paper/2212.08930/) (2022)
- [FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization](https://mlsanity.com/paper/2206.03966/) (2022)
- [FEATHERS: Federated Architecture and Hyperparameter Search](https://mlsanity.com/paper/2206.12342/) (2022)
- [FedAVO: Improving Communication Efficiency in Federated Learning with African Vultures Optimizer](https://mlsanity.com/paper/2305.01154/) (2023)
- [Federated Learning Hyper-Parameter Tuning from a System Perspective](https://mlsanity.com/paper/2211.13656/) (2022)
- [FedPop: Federated Population-based Hyperparameter Tuning](https://mlsanity.com/paper/2308.08634/) (2023)
- [SODA: Robust Training of Test-Time Data Adaptors](https://mlsanity.com/paper/2310.11093/) (2023)
- [DistDD: Distributed Data Distillation Aggregation through Gradient Matching](https://mlsanity.com/paper/2410.08665/) (2024)
- [Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism](https://mlsanity.com/paper/2411.12244/) (2024)
- [Toward Theoretical Guidance for Two Common Questions in Practical Cross-Validation based Hyperparameter Selection](https://mlsanity.com/paper/2301.05131/) (2023)

## Abstract (arXiv)

We address the relatively unexplored problem of hyper-parameter optimization (HPO) for federated learning (FL-HPO). We introduce Federated Loss suRface Aggregation (FLoRA), the first FL-HPO solution framework that can address use cases of tabular data and gradient boosting training algorithms in addition to stochastic gradient descent/neural networks commonly addressed in the FL literature. The framework enables single-shot FL-HPO, by first identifying a good set of hyper-parameters that are used in a **single** FL training. Thus, it enables FL-HPO solutions with minimal additional communication overhead compared to FL training without HPO. Our empirical evaluation of FLoRA for Gradient Boosted Decision Trees on seven OpenML data sets demonstrates significant model accuracy improvements over the considered baseline, and robustness to increasing number of parties involved in FL-HPO training.