mlsanityData to 7 Oct 2026

Revisiting Weighted Aggregation in Federated Learning with Neural Networks

arXiv 2302.10911 ↗First posted 14 Feb 2023cs.LGcited by 57

Not on the map. Why?

Abstract (arXiv)

In federated learning (FL), weighted aggregation of local models is conducted to generate a global model, and the aggregation weights are normalized (the sum of weights is 1) and proportional to the local data sizes. In this paper, we revisit the weighted aggregation process and gain new insights into the training dynamics of FL. First, we find that the sum of weights can be smaller than 1, causing global weight shrinking effect (analogous to weight decay) and improving generalization. We explore how the optimal shrinking factor is affected by clients' data heterogeneity and local epochs. Second, we dive into the relative aggregation weights among clients to depict the clients' importance. We develop client coherence to study the learning dynamics and find a critical point that exists. Before entering the critical point, more coherent clients play more essential roles in generalization. Based on the above insights, we propose an effective method for Federated Learning with Learnable Aggregation Weights, named as FedLAW. Extensive experiments verify that our method can improve the generalization of the global model by a large margin on different datasets and models.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 48 of its 67 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

48 of 67 references.

References · 48

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  9. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  11. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  12. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
36 more
  1. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  2. 2020Federated Learning with Matched Averagingcited by 564
  3. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  4. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  6. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  7. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  8. 2021A Field Guide to Federated Optimizationcited by 312
  9. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  10. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  11. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  12. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  13. 2019Model Fusion via Optimal Transportcited by 217
  14. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  15. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  16. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  17. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  18. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  19. 2021Distributionally Robust Federated Averagingcited by 104
  20. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  21. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  22. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  23. 2021On Large-Cohort Training for Federated Learningcited by 73
  24. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  25. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  26. 2020Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimizationcited by 41
  27. 2020On the training dynamics of deep networks with L2 regularizationcited by 38
  28. 2019Hyp-RL : Hyperparameter Optimization by Reinforcement Learningcited by 36
  29. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  30. 2022Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matchingcited by 24
  31. 2021Auto-FedAvg: Learnable Federated Averaging for Multi-Institutional Medical Image Segmentationcited by 19
  32. 2019Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameterscited by 16
  33. 2022Auto-FedRL: Federated Hyperparameter Optimization for Multi-institutional Medical Image Segmentationcited by 16
  34. 2021Critical Learning Periods in Federated Learningcited by 8
  35. 2020Making Coherence Out of Nothing At All: Measuring the Evolution of Gradient Alignmentcited by 7
  36. 2022DRFLM: Distributionally Robust Federated Learning with Inter-client Noise via Local Mixupcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 14; 2025: 21; 2026: 9.

57 papers.

Sort:

By area:

  1. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  2. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  3. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 66
  4. 2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 64
  5. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  6. 2023Think Twice Before Selection: Federated Evidential Active Learning for Medical Image Analysis with Domain Shiftscited by 26
  7. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
  8. 2025LEO-Split: A Semi-Supervised Split Learning Framework over LEO Satellite Networkscited by 22
  9. 2025FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectorscited by 19
  10. 2023GPFL: Simultaneously Learning Global and Personalized Feature Information for Personalized Federated Learningcited by 18
  11. 2023MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulationcited by 13
  12. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
38 more
  1. 2024AFL: A Single-Round Analytic Approach for Federated Learning with Pre-trained Modelscited by 11
  2. 2025Geometric Knowledge-Guided Localized Global Distribution Alignment for Federated Learningcited by 11
  3. 2025MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Userscited by 10
  4. 2023Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Datacited by 7
  5. 2023Window-based Model Averaging Improves Generalization in Heterogeneous Federated Learningcited by 7
  6. 2024Improving Model Fusion by Training-time Neuron Alignment with Fixed Neuron Anchorscited by 7
  7. 2025Subgraph Federated Learning for Local Generalizationcited by 6
  8. 2025FedLWS: Federated Learning with Adaptive Layer-wise Weight Shrinkingcited by 6
  9. 2023FediOS: Decoupling Orthogonal Subspaces for Personalization in Feature-skew Federated Learningcited by 5
  10. 2024FedCAda: Adaptive Client-Side Optimization for Accelerated and Stable Federated Learningcited by 5
  11. 2024Navigating Distribution Shifts in Medical Image Analysis: A Surveycited by 5
  12. 2025You Are Your Own Best Teacher: Achieving Centralized-level Performance in Federated Learning under Heterogeneous and Long-tailed Datacited by 5
  13. 2023StoCFL: A Stochastically Clustered Federated Learning Framework for Non-IID Data with Dynamic Client Participationcited by 4
  14. 2023FedNAR: Federated Optimization with Normalized Annealing Regularizationcited by 4
  15. 2025FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learningcited by 4
  16. 2024SignSGD with Federated Votingcited by 2
  17. 2024Dataset Distillation-based Hybrid Federated Learning on Non-IID Datacited by 2
  18. 2024Overcoming label shift with target-aware federated learningcited by 2
  19. 2024Election of Collaborators via Reinforcement Learning for Federated Brain Tumor Segmentationcited by 2
  20. 2025Toward a Holistic Approach to Continual Model Mergingcited by 2
  21. 2024SignSGD with Federated Defense: Harnessing Adversarial Attacks through Gradient Sign Decodingcited by 1
  22. 2024CollabEdit: Towards Non-destructive Collaborative Knowledge Editingcited by 1
  23. 2024Information-Geometric Barycenters for Bayesian Federated Learningcited by 1
  24. 2025Carpe Diem: Critical Learning Period-Aware Contract-Based Incentives for Federated Learningcited by 1
  25. 2025Decentralized Dynamic Cooperation of Personalized Models for Federated Continual Learningcited by 1
  26. 2025Distributionally Robust Federated Learning with Outlier Resiliencecited by 1
  27. 2025FedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learningcited by 1
  28. 2026Distribution Alignment for One-Shot Federated Learning via Optimal Transportcited by 1
  29. 2023Tackling Intertwined Data and Device Heterogeneities in Federated Learning with Unlimited Stalenesscited by 0
  30. 2025Encoded Spatial Attribute in Multi-Tier Federated Learningcited by 0
  31. 2025FedVSR: Towards Model-Agnostic Federated Learning in Video Super-Resolutioncited by 0
  32. 2025Communication Efficient Federated Learning with Linear Convergence on Heterogeneous Datacited by 0
  33. 2025Local Data Quantity-Aware Weighted Averaging for Federated Learning with Dishonest Clientscited by 0
  34. 2025DRAGD: A Federated Unlearning Data Reconstruction Attack Based on Gradient Differencescited by 0
  35. 2025FIDELIS: Blockchain-Enabled Protection Against Poisoning Attacks in Federated Learningcited by 0
  36. 2025Generalizable Federated Learning using Client Adaptive Focal Modulationcited by 0
  37. 2025Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistencycited by 0
  38. 2026Towards Performance-Enhanced Model-Contrastive Federated Learning using Historical Information in Heterogeneous Scenarioscited by 0

50 most cited of 57 All 57 (JSON)