How Much Regularization Survives Averaging? Update Masking in Federated Learning
Not on the map. Why?
Abstract (arXiv)
Federated learning on non-IID data seeks flat minima to generalize across clients, and existing methods borrow sharpness-aware minimization from centralized training. There is a second way to reach flat minima, in which the regularization comes for free from noise added to the parameter updates, and it has never been carried over to the federated setting as an implicit regularizer. We show the reason. Masking charges the optimizer for moving in sharp directions. We prove that when each client draws its own mask, federated averaging weakens that charge by exactly the cohort size, and that giving every client the same mask brings it back by a factor equal to the inverse gradient diversity of the cohort. In our experiment setting on CIFAR-10, that factor is 1.19 out of a possible 10. Turning off minibatch sampling raises it to 8.96, while changing data heterogeneity a thousandfold leaves it between 1.17 and 1.50. The configurations keeping the regularization train far too poorly to use.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 11 references that are in the map. First posted on arXiv 24 Aug 2026.
Builds on
7 of 11 references.
References · 7
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
- 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 69
- 2021Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devicescited by 56
- 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
- 2022Differentially Private Federated Learning with Local Regularization and Sparsificationcited by 36
- 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7
Built on by
Not cited yet.