Data to 7 Oct 2026

Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization

arXiv 2007.07481 ↗First posted 15 Jul 2020cs.LG cs.DC stat.MLcited by 897

Not on the map. Why?

Abstract (arXiv)

In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such as FedAvg and FedProx and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.

Key facts. Cited by 897 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 78; 2022: 161; 2023: 191; 2024: 195; 2025: 161; 2026: 111.

897 papers.

Sort:

By area:

2021Model-Contrastive Federated Learningcited by 762
2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
2021Federated Learning on Non-IID Data: A Surveycited by 345
2021A Field Guide to Federated Optimizationcited by 312
2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
2021Federated Multi-Task Learning under a Mixture of Distributionscited by 189
2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181

38 more

2022NOTE: Robust Continual Test-time Adaptation Against Temporal Correlationcited by 150
2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 121
2021FairFed: Enabling Group Fairness in Federated Learningcited by 114
2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
2021DENSE: Data-Free One-Shot Federated Learningcited by 100
2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
2022Federated Learning on Non-IID Graphs via Structural Knowledge Sharingcited by 88
2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
2021FedProc: Prototypical Contrastive Federated Learning on Non-IID datacited by 80
2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
2021FedCM: Federated Learning with Client-level Momentumcited by 77
2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 68
2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 66
2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 66
2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 65
2021LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learningcited by 63
2022Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Featurescited by 63
2021HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Imagescited by 62
2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62
2022FedCorr: Multi-Stage Federated Learning for Label Noise Correctioncited by 61
2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
2022Communication-Efficient Adaptive Federated Learningcited by 55
2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53

50 most cited of 897 All 897 (JSON)