mlsanityData to 7 Oct 2026

FedExP: Speeding Up Federated Averaging via Extrapolation

arXiv 2301.09604 ↗First posted 23 Jan 2023cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

Federated Averaging (FedAvg) remains the most popular algorithm for Federated Learning (FL) optimization due to its simple implementation, stateless nature, and privacy guarantees combined with secure aggregation. Recent work has sought to generalize the vanilla averaging in FedAvg to a generalized gradient descent step by treating client updates as pseudo-gradients and using a server step size. While the use of a server step size has been shown to provide performance improvement theoretically, the practical benefit of the server step size has not been seen in most existing works. In this work, we present FedExP, a method to adaptively determine the server step size in FL based on dynamically varying pseudo-gradients throughout the FL process. We begin by considering the overparameterized convex regime, where we reveal an interesting similarity between FedAvg and the Projection Onto Convex Sets (POCS) algorithm. We then show how FedExP can be motivated as a novel extension to the extrapolation mechanism that is used to speed up POCS. Our theoretical analysis later also discusses the implications of FedExP in underparameterized and non-convex settings. Experimental results show that FedExP consistently converges faster than FedAvg and competing baselines on a range of realistic FL datasets.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 23 Jan 2023.

Builds on

33 of 55 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  10. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  11. 2018LEAF: A Benchmark for Federated Settingscited by 780
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
21 more
  1. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  2. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  3. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  4. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  5. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  6. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  7. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 150
  8. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  9. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  10. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  11. 2019Adaptive Gradient Descent without Descentcited by 79
  12. 2020FastSecAgg: Scalable Secure Aggregation for Privacy-Preserving Federated Learningcited by 60
  13. 2019Revisiting the Polyak step sizecited by 55
  14. 2020On the Outsized Importance of Learning Rates in Local Update Methodscited by 34
  15. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  16. 2022Server-Side Stepsizes and Sampling Without Replacement Provably Help in Federated Optimizationcited by 23
  17. 2020AdaScale SGD: A User-Friendly Algorithm for Distributed Trainingcited by 16
  18. 2021Neural Tangent Kernel Empowered Federated Learningcited by 15
  19. 2019Stabilized Barzilai-Borwein methodcited by 10
  20. 2021Local SGD Optimizes Overparameterized Neural Networks in Polynomial Timecited by 10
  21. 2022Adaptive Learning Rates for Faster Stochastic Gradient Methodscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 20; 2025: 22; 2026: 10.

68 papers.

Sort:

By area:

  1. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  2. 2023Deep Model Fusion: A Surveycited by 89
  3. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  4. 2025Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretrainingcited by 16
  5. 2023Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombinationcited by 13
  6. 2024The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communicationcited by 12
  7. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  8. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  9. 2024Tackling Noisy Clients in Federated Learning with End-to-end Label Correctioncited by 11
  10. 2026MM-OpenFGL: A Comprehensive Benchmark for Multimodal Federated Graph Learningcited by 11
  11. 2024FADAS: Towards Federated Adaptive Asynchronous Optimizationcited by 10
  12. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
38 more
  1. 2023FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descentcited by 8
  2. 2023Handling Data Heterogeneity via Architectural Design for Federated Visual Recognitioncited by 8
  3. 2023Federated Learning with Manifold Regularization and Normalized Update Reaggregationcited by 8
  4. 2023Federated Learning While Providing Model as a Service: Joint Training and Inference Optimizationcited by 8
  5. 2024Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Driftcited by 8
  6. 2023FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learningcited by 7
  7. 2025Federated Domain Generalization with Data-free On-server Matching Gradientcited by 7
  8. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 7
  9. 2023Fake It Till Make It: Federated Learning with Consensus-Oriented Generationcited by 6
  10. 2025Federated Learning with Sample-level Client Drift Mitigationcited by 6
  11. 2025FNBench: Benchmarking Robust Federated Learning against Noisy Labelscited by 6
  12. 2023Distortion Resilience for Goal-Oriented Semantic Communicationcited by 4
  13. 2023FedNAR: Federated Optimization with Normalized Annealing Regularizationcited by 4
  14. 2024Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspectivecited by 4
  15. 2025FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learningcited by 4
  16. 2023Cooperative Hardware-Prompt Learning for Snapshot Compressive Imagingcited by 3
  17. 2023FedBug: A Bottom-Up Gradual Unfreezing Framework for Federated Learningcited by 3
  18. 2023Federated Learning Empowered by Generative Contentcited by 3
  19. 2024On the Convergence of FedProx with Extrapolation and Inexact Proxcited by 3
  20. 2024Towards Layer-Wise Personalized Federated Learning: Adaptive Layer Disentanglement via Conflicting Gradientscited by 3
  21. 2024Federated Communication-Efficient Multi-Objective Optimizationcited by 3
  22. 2024FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learningcited by 2
  23. 2024Weight Scope Alignment: A Frustratingly Easy Method for Model Mergingcited by 2
  24. 2024FedECADO: A Dynamical System Model of Federated Learningcited by 2
  25. 2026FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewardscited by 2
  26. 2024FAGH: Accelerating Federated Learning with Approximated Global Hessiancited by 1
  27. 2024Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learningcited by 1
  28. 2024Tighter Performance Theory of FedExProxcited by 1
  29. 2024Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Modelscited by 1
  30. 2025Accelerating Differentially Private Federated Learning via Adaptive Extrapolationcited by 1
  31. 2025FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCAcited by 1
  32. 2025FedMeNF: Privacy-Preserving Federated Meta-Learning for Neural Fieldscited by 1
  33. 2023Improving Federated Aggregation with Deep Unfolding Networkscited by 0
  34. 2023Evaluating Multi-Global Server Architecture for Federated Learningcited by 0
  35. 2023AIJack: Let's Hijack AI! Security and Privacy Risk Simulator for Machine Learningcited by 0
  36. 2024Painless Federated Learning: An Interplay of Line-Search and Extrapolationcited by 0
  37. 2024fluke: Federated Learning Utility frameworK for Experimentation and researchcited by 0
  38. 2025A Randomized Zeroth-Order Hierarchical Framework for Heterogeneous Federated Learningcited by 0

50 most cited of 68 All 68 (JSON)