Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optima
Not on the map. Why?
Abstract (arXiv)
Federated optimization is a constrained form of distributed optimization that enables training a global model without directly sharing client data. Although existing algorithms can guarantee convergence in theory and often achieve stable training in practice, the reasons behind performance degradation under data heterogeneity remain unclear. To address this gap, the main contribution of this paper is to provide a theoretical perspective that explains why such degradation occurs. We introduce the assumption that heterogeneous client data lead to distinct local optima, and show that this assumption implies two key consequences: 1) the distance among clients' local optima raises the lower bound of the global objective, making perfect fitting of all client data impossible; and 2) in the final training stage, the global model oscillates within a region instead of converging to a single optimum, limiting its ability to fully fit the data. These results provide a principled explanation for performance degradation in non-iid settings, which we further validate through experiments across multiple tasks and neural network architectures. The framework used in this paper is open-sourced at: https://github.com/NPCLEI/fedtorch.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 47 references that are in the map. First posted on arXiv 1 Nov 2025.
Builds on
29 of 47 references.
References · 29
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2020Adaptive Federated Optimizationcited by 1,036
- 2020A Simple Convergence Proof of Adam and Adagradcited by 169
- 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
- 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
- 2022Vertical Federated Learning: Concepts, Advances and Challengescited by 104
17 more
- 2023Convergence of Adam Under Relaxed Assumptionscited by 97
- 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 68
- 2022Communication-Efficient Adaptive Federated Learningcited by 55
- 2019Understanding the Role of Momentum in Stochastic Gradient Methodscited by 42
- 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
- 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
- 2020On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settingscited by 27
- 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
- 2023On the Role of Server Momentum in Federated Learningcited by 16
- 2020Federated Learning with Nesterov Accelerated Gradientcited by 11
- 2024Federated Optimization with Doubly Regularized Drift Correctioncited by 10
- 2024FADAS: Towards Federated Adaptive Asynchronous Optimizationcited by 10
- 2023Adaptive Federated Learning with Auto-Tuned Clientscited by 7
- 2023Locally Adaptive Federated Learningcited by 7
- 2024Stabilized Proximal-Point Methods for Federated Optimizationcited by 6
- 2023FedNAR: Federated Optimization with Normalized Annealing Regularizationcited by 4
- 2024A New Theoretical Perspective on Data Heterogeneity in Federated Optimizationcited by 4
Built on by
Not cited yet.