mlsanityData to 7 Oct 2026

FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learning

arXiv 2207.14130 ↗First posted 28 Jul 2022cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

Data-heterogeneous federated learning (FL) systems suffer from two significant sources of convergence error: 1) client drift error caused by performing multiple local optimization steps at clients, and 2) partial client participation error caused by the fact that only a small subset of the edge clients participate in every training round. We find that among these, only the former has received significant attention in the literature. To remedy this, we propose FedVARP, a novel variance reduction algorithm applied at the server that eliminates error due to partial client participation. To do so, the server simply maintains in memory the most recent update for each client and uses these as surrogate updates for the non-participating clients in every round. Further, to alleviate the memory requirement at the server, we propose a novel clustering-based variance reduction algorithm ClusterFedVARP. Unlike previously proposed methods, both FedVARP and ClusterFedVARP do not require additional computation at clients or communication of additional optimization parameters. Through extensive experiments, we show that FedVARP outperforms state-of-the-art methods, and ClusterFedVARP achieves performance comparable to FedVARP with much less memory requirements.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 39 of its 45 references that are in the map. First posted on arXiv 28 Jul 2022.

Builds on

39 of 45 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  9. 2018LEAF: A Benchmark for Federated Settingscited by 780
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  11. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  12. 2021Federated Learning Based on Dynamic Regularizationcited by 555
27 more
  1. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  2. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  3. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  4. 2021A Field Guide to Federated Optimizationcited by 312
  5. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  6. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  7. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 283
  8. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  9. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  10. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  11. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  12. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  13. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  14. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  15. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
  16. 2020Optimal Client Sampling for Federated Learningcited by 123
  17. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  18. 2018Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithmscited by 106
  19. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  20. 2017Non-convex Finite-Sum Optimization Via SCSG Methodscited by 61
  21. 2020Faster Non-Convex Federated Learning via Global and Local Momentumcited by 48
  22. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  23. 2015Variance Reduced Stochastic Gradient Descent with Neighborscited by 39
  24. 2019Hybrid Stochastic Gradient Descent Algorithms for Stochastic Nonconvex Optimizationcited by 31
  25. 2021ZeroSARAH: Efficient Nonconvex Finite-Sum Optimization with Zero Full Gradient Computationcited by 28
  26. 2016Fast Incremental Method for Nonconvex Optimizationcited by 9
  27. 2016Exploiting the Structure: Stochastic Gradient Methods Using Raw Clusterscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 17; 2024: 18; 2025: 15; 2026: 10.

62 papers.

Sort:

By area:

  1. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  2. 2024Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domainscited by 27
  3. 2024Advances in APPFL: A Comprehensive and Extensible Federated Learning Frameworkcited by 19
  4. 2023Accelerating Hybrid Federated Learning Convergence under Partial Participationcited by 15
  5. 2023Feature Matching Data Synthesis for Non-IID Federated Learningcited by 13
  6. 2024Byzantine-Robust Federated Learning: Impact of Client Subsampling and Local Updatescited by 13
  7. 2024Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailabilitycited by 9
  8. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  9. 2023Federated Minimax Optimization with Client Heterogeneitycited by 8
  10. 2023Delay Sensitive Hierarchical Federated Learning with Stochastic Local Updatescited by 8
  11. 2023On the Local Cache Update Rules in Streaming Federated Learningcited by 8
  12. 2022To Store or Not? Online Data Selection for Federated Learning with Limited Storagecited by 7
38 more
  1. 2023Towards Bias Correction of FedAvg over Nonuniform and Time-Varying Communicationscited by 6
  2. 2025Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainablecited by 6
  3. 2023Federated Learning in the Presence of Adversarial Client Unavailabilitycited by 5
  4. 2024FedStale: leveraging stale client updates in federated learningcited by 5
  5. 2024DRACO: Decentralized Asynchronous Federated Learning over Row-Stochastic Wireless Networkscited by 5
  6. 2023Entropy-driven Fair and Effective Federated Learningcited by 4
  7. 2023Stochastic Smoothed Gradient Descent Ascent for Federated Minimax Optimizationcited by 4
  8. 2025FedCanon: Non-Convex Composite Federated Learning with Efficient Proximal Operation on Heterogeneous Datacited by 4
  9. 2023TAMUNA: Doubly Accelerated Distributed Optimization under Partial Participationcited by 3
  10. 2023Stabilizing and Improving Federated Learning with Non-IID Data and Client Dropoutcited by 3
  11. 2024Empowering Federated Learning with Implicit Gossiping: Mitigating Connection Unreliability Amidst Unknown and Arbitrary Dynamicscited by 3
  12. 2024FedAST: Federated Asynchronous Simultaneous Trainingcited by 3
  13. 2024Federated Communication-Efficient Multi-Objective Optimizationcited by 3
  14. 2025Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participationcited by 3
  15. 2026Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluationscited by 3
  16. 2023Controlling Federated Learning for Covertnesscited by 2
  17. 2024A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needscited by 2
  18. 2025GC-Fed: Gradient Centralized Federated Learning with Partial Client Participationcited by 2
  19. 2023A Lightweight Method for Tackling Unknown Participation Statistics in Federated Averagingcited by 1
  20. 2023Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variancecited by 1
  21. 2024Fair Federated Data Clustering through Personalization: Bridging the Gap between Diverse Data Distributionscited by 1
  22. 2024Sequential Federated Learning in Hierarchical Architecture on Non-IID Datasetscited by 1
  23. 2024Debiasing Federated Learning with Correlated Client Participationcited by 1
  24. 2024Communication-Efficient Federated Learning under Dynamic Device Arrival and Departure: Convergence Analysis and Algorithm Designcited by 1
  25. 2025Heterogeneity-Aware Client Sampling for Optimal and Efficient Federated Learningcited by 1
  26. 2025Helmsman: Autonomous Synthesis of Federated Learning Systems via Collaborative LLM Agentscited by 1
  27. 2022Versatile Single-Loop Method for Gradient Estimator: First and Second Order Optimality, and its Application to Federated Learningcited by 0
  28. 2023Tackling Computational Heterogeneity in FL: A Few Theoretical Insightscited by 0
  29. 2023On the Convergence of Federated Averaging under Partial Participation for Over-parameterized Neural Networkscited by 0
  30. 2024FedShift: Robust Federated Learning Aggregation Scheme in Resource Constrained Environment via Weight Shiftingcited by 0
  31. 2024FedAR: Addressing Client Unavailability in Federated Learning with Local Update Approximation and Rectificationcited by 0
  32. 2024Convergent Differential Privacy Analysis for General Federated Learningcited by 0
  33. 2024Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimizationcited by 0
  34. 2025Extra Clients at No Extra Cost: Overcome Data Heterogeneity in Federated Learning with Filter Decompositioncited by 0
  35. 2025Unifying Physics- and Data-Driven Modeling via Novel Causal Spatiotemporal Graph Neural Network for Interpretable Epidemic Forecastingcited by 0
  36. 2025Personalized Federated Learning under Model Dissimilarity Constraintscited by 0
  37. 2025Accelerated Training of Federated Learning via Second-Order Methodscited by 0
  38. 2025Overcoming Challenges of Partial Client Participation in Federated Learning : A Comprehensive Reviewcited by 0

50 most cited of 62 All 62 (JSON)