# Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameters

arXiv: https://arxiv.org/abs/1912.13075 · HTML: https://mlsanity.com/paper/1912.13075/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2019.

## Built on by (most cited first, 16 of 16)

- [Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing](https://mlsanity.com/paper/2106.04502/) (2021)
- [Revisiting Weighted Aggregation in Federated Learning with Neural Networks](https://mlsanity.com/paper/2302.10911/) (2023)
- [Fed2: Feature-Aligned Federated Learning](https://mlsanity.com/paper/2111.14248/) (2021)
- [Auto-FedRL: Federated Hyperparameter Optimization for Multi-institutional Medical Image Segmentation](https://mlsanity.com/paper/2203.06338/) (2022)
- [FLoRA: Single-shot Hyper-parameter Optimization for Federated Learning](https://mlsanity.com/paper/2112.08524/) (2021)
- [On Noisy Evaluation in Federated Hyperparameter Tuning](https://mlsanity.com/paper/2212.08930/) (2022)
- [CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning](https://mlsanity.com/paper/2204.01542/) (2022)
- [FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization](https://mlsanity.com/paper/2206.03966/) (2022)
- [Single-shot Hyper-parameter Optimization for Federated Learning: A General Algorithm & Analysis](https://mlsanity.com/paper/2202.08338/) (2022)
- [FEATHERS: Federated Architecture and Hyperparameter Search](https://mlsanity.com/paper/2206.12342/) (2022)
- [Federated Hypergradient Descent](https://mlsanity.com/paper/2211.02106/) (2022)
- [Federated Automatic Differentiation](https://mlsanity.com/paper/2301.07806/) (2023)
- [Federated Learning Hyper-Parameter Tuning from a System Perspective](https://mlsanity.com/paper/2211.13656/) (2022)
- [FedPop: Federated Population-based Hyperparameter Tuning](https://mlsanity.com/paper/2308.08634/) (2023)
- [Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism](https://mlsanity.com/paper/2411.12244/) (2024)
- [AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environments](https://mlsanity.com/paper/2606.08197/) (2026)

## Abstract (arXiv)

Federated learning is a distributed, privacy-aware learning scenario which trains a single model on data belonging to several clients. Each client trains a local model on its data and the local models are then aggregated by a central party. Current federated learning methods struggle in cases with heterogeneous client-side data distributions which can quickly lead to divergent local models and a collapse in performance. Careful hyper-parameter tuning is particularly important in these cases but traditional automated hyper-parameter tuning methods would require several training trials which is often impractical in a federated learning setting. We describe a two-pronged solution to the issues of robustness and hyper-parameter tuning in federated learning settings. We propose a novel representation matching scheme that reduces the divergence of local models by ensuring the feature representations in the global (aggregate) model can be derived from the locally learned representations. We also propose an online hyper-parameter tuning scheme which uses an online version of the REINFORCE algorithm to find a hyper-parameter distribution that maximizes the expected improvements in training loss. We show on several benchmarks that our two-part scheme of local representation matching and global adaptive hyper-parameters significantly improves performance and training robustness.