mlsanityData to 7 Oct 2026

A Non-parametric View of FedAvg and FedProx: Beyond Stationary Points

arXiv 2106.15216 ↗First posted 29 Jun 2021stat.ML cs.DC cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is a promising decentralized learning framework and has great potentials in privacy preservation and in lowering the computation load at the cloud. Recent work showed that FedAvg and FedProx - the two widely-adopted FL algorithms - fail to reach the stationary points of the global optimization objective even for homogeneous linear regression problems. Further, it is concerned that the common model learned might not generalize well locally at all in the presence of heterogeneity. In this paper, we analyze the convergence and statistical efficiency of FedAvg and FedProx, addressing the above two concerns. Our analysis is based on the standard non-parametric regression in a reproducing kernel Hilbert space (RKHS), and allows for heterogeneous local data distributions and unbalanced local datasets. We prove that the estimation errors, measured in either the empirical norm or the RKHS norm, decay with a rate of 1/t in general and exponentially for finite-rank kernels. In certain heterogeneous settings, these upper bounds also imply that both FedAvg and FedProx achieve the optimal error rate. To further analytically quantify the impact of the heterogeneity at each client, we propose and characterize a novel notion-federation gain, defined as the reduction of the estimation error for a client to join the FL. We discover that when the data heterogeneity is moderate, a client with limited local data can benefit from a common model with a large federation gain. Numerical experiments further corroborate our theoretical findings.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 21 of its 44 references that are in the map. First posted on arXiv 29 Jun 2021.

Builds on

21 of 44 references.

References · 21

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2018Federated Learning with Non-IID Datacited by 1,099
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  8. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  11. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  12. 2020Adaptive Personalized Federated Learningcited by 353
9 more
  1. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  2. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
  3. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  4. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  5. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  6. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 86
  7. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  8. 2019Exact minimax risk for linear least squares, and the lower tail of sample covariance matricescited by 34
  9. 2020On the Outsized Importance of Learning Rates in Local Update Methodscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 5; 2025: 4; 2026: 3.

13 papers.

Sort:

  1. 2022Global Convergence of Federated Learning for Mixed Regressioncited by 6
  2. 2024FedNS: A Fast Sketching Newton-Type Algorithm for Federated Learningcited by 3
  3. 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environmentscited by 2
  4. 2024A Kernel Perspective on Distillation-based Collaborative Learningcited by 1
  5. 2026Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gapscited by 1
  6. 2026FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projectioncited by 1
  7. 2024Enhancing IoT Security Against DDoS Attacks through Federated Learningcited by 0
  8. 2024FLeNS: Federated Learning with Enhanced Nesterov-Newton Sketchcited by 0
  9. 2025FedVSR: Towards Model-Agnostic Federated Learning in Video Super-Resolutioncited by 0
  10. 2025Traversal Learning: A Lossless And Efficient Distributed Learning Frameworkcited by 0
  11. 2025FedP3E: Privacy-Preserving Prototype Exchange for Non-IID IoT Malware Detection in Cross-Silo Federated Learningcited by 0
  12. 2025A Robust Clustered Federated Learning Approach for Non-IID Data with Quantity Skewcited by 0
1 more
  1. 2026Decision-Focused Federated Learning Under Heterogeneous Objectives and Constraintscited by 0