# Revisiting Weighted Aggregation in Federated Learning with Neural Networks

arXiv: https://arxiv.org/abs/2302.10911 · HTML: https://mlsanity.com/paper/2302.10911/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 67 references that are in the map.
- First posted on arXiv 14 Feb 2023.

## Builds on (most cited first, 30 of 48)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Personalized Cross-Silo Federated Learning on Non-IID Data](https://mlsanity.com/paper/2007.03797/) (2020)
- [Forward and Reverse Gradient-Based Hyperparameter Optimization](https://mlsanity.com/paper/1703.01785/) (2017)
- [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [The Break-Even Point on Optimization Trajectories of Deep Neural Networks](https://mlsanity.com/paper/2002.09572/) (2020)
- [Efficient Sharpness-aware Minimization for Improved Training of Neural Networks](https://mlsanity.com/paper/2110.03141/) (2021)
- [An Improved Analysis of Training Over-parameterized Deep Neural Networks](https://mlsanity.com/paper/1906.04688/) (2019)
- [Three Mechanisms of Weight Decay Regularization](https://mlsanity.com/paper/1810.12281/) (2018)

## Built on by (most cited first, 30 of 58)

- [OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning](https://mlsanity.com/paper/2402.06954/) (2024)
- [Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark](https://mlsanity.com/paper/2311.06750/) (2023)
- [FedDisco: Federated Learning with Discrepancy-Aware Collaboration](https://mlsanity.com/paper/2305.19229/) (2023)
- [TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation](https://mlsanity.com/paper/2303.06937/) (2023)
- [No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier](https://mlsanity.com/paper/2303.10058/) (2023)
- [FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models](https://mlsanity.com/paper/2406.04845/) (2024)
- [Think Twice Before Selection: Federated Evidential Active Learning for Medical Image Analysis with Domain Shifts](https://mlsanity.com/paper/2312.02567/) (2023)
- [LEO-Split: A Semi-Supervised Split Learning Framework over LEO Satellite Networks](https://mlsanity.com/paper/2501.01293/) (2025)
- [FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectors](https://mlsanity.com/paper/2503.15842/) (2025)
- [GPFL: Simultaneously Learning Global and Personalized Feature Information for Personalized Federated Learning](https://mlsanity.com/paper/2308.10279/) (2023)
- [MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulation](https://mlsanity.com/paper/2311.13348/) (2023)
- [Task Arithmetic Through The Lens Of One-Shot Federated Learning](https://mlsanity.com/paper/2411.18607/) (2024)
- [Geometric Knowledge-Guided Localized Global Distribution Alignment for Federated Learning](https://mlsanity.com/paper/2503.06457/) (2025)
- [AFL: A Single-Round Analytic Approach for Federated Learning with Pre-trained Models](https://mlsanity.com/paper/2405.16240/) (2024)
- [MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users](https://mlsanity.com/paper/2502.02982/) (2025)
- [Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data](https://mlsanity.com/paper/2305.07845/) (2023)
- [Window-based Model Averaging Improves Generalization in Heterogeneous Federated Learning](https://mlsanity.com/paper/2310.01366/) (2023)
- [Improving Model Fusion by Training-time Neuron Alignment with Fixed Neuron Anchors](https://mlsanity.com/paper/2402.01342/) (2024)
- [FedLWS: Federated Learning with Adaptive Layer-wise Weight Shrinking](https://mlsanity.com/paper/2503.15111/) (2025)
- [Subgraph Federated Learning for Local Generalization](https://mlsanity.com/paper/2503.03995/) (2025)
- [FediOS: Decoupling Orthogonal Subspaces for Personalization in Feature-skew Federated Learning](https://mlsanity.com/paper/2311.18559/) (2023)
- [FedCAda: Adaptive Client-Side Optimization for Accelerated and Stable Federated Learning](https://mlsanity.com/paper/2405.11811/) (2024)
- [Navigating Distribution Shifts in Medical Image Analysis: A Survey](https://mlsanity.com/paper/2411.05824/) (2024)
- [You Are Your Own Best Teacher: Achieving Centralized-level Performance in Federated Learning under Heterogeneous and Long-tailed Data](https://mlsanity.com/paper/2503.06916/) (2025)
- [StoCFL: A Stochastically Clustered Federated Learning Framework for Non-IID Data with Dynamic Client Participation](https://mlsanity.com/paper/2303.00897/) (2023)
- [FedNAR: Federated Optimization with Normalized Annealing Regularization](https://mlsanity.com/paper/2310.03163/) (2023)
- [FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2510.20250/) (2025)
- [Dataset Distillation-based Hybrid Federated Learning on Non-IID Data](https://mlsanity.com/paper/2409.17517/) (2024)
- [Toward a Holistic Approach to Continual Model Merging](https://mlsanity.com/paper/2509.23592/) (2025)
- [SignSGD with Federated Voting](https://mlsanity.com/paper/2403.16372/) (2024)

## Abstract (arXiv)

In federated learning (FL), weighted aggregation of local models is conducted to generate a global model, and the aggregation weights are normalized (the sum of weights is 1) and proportional to the local data sizes. In this paper, we revisit the weighted aggregation process and gain new insights into the training dynamics of FL. First, we find that the sum of weights can be smaller than 1, causing global weight shrinking effect (analogous to weight decay) and improving generalization. We explore how the optimal shrinking factor is affected by clients' data heterogeneity and local epochs. Second, we dive into the relative aggregation weights among clients to depict the clients' importance. We develop client coherence to study the learning dynamics and find a critical point that exists. Before entering the critical point, more coherent clients play more essential roles in generalization. Based on the above insights, we propose an effective method for Federated Learning with Learnable Aggregation Weights, named as FedLAW. Extensive experiments verify that our method can improve the generalization of the global model by a large margin on different datasets and models.