mlsanityData to 7 Oct 2026

Federated Optimization: Distributed Machine Learning for On-Device Intelligence

arXiv 1610.02527 ↗First posted 8 Oct 2016cs.LGcited by 558

Not on the map. Why?

Abstract (arXiv)

We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are unevenly distributed over an extremely large number of nodes. The goal is to train a high-quality centralized model. We refer to this setting as Federated Optimization. In this setting, communication efficiency is of the utmost importance and minimizing the number of rounds of communication is the principal goal. A motivating example arises when we keep the training data locally on users' mobile devices instead of logging it to a data center for training. In federated optimziation, the devices are used as compute nodes performing computation on their local data in order to update a global model. We suppose that we have extremely large number of devices in the network --- as many as the number of users of a given service, each of which has only a tiny fraction of the total data available. In particular, we expect the number of data points available locally to be much smaller than the number of devices. Additionally, since different users generate data with different patterns, it is reasonable to assume that no device has a representative sample of the overall distribution. We show that existing algorithms are not suitable for this setting, and propose a new algorithm which shows encouraging experimental results for sparse convex problems. This work also sets a path for future research needed in the context of federated optimization.

Key facts. Cited by 558 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Oct 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 121; 2022: 120; 2023: 126; 2024: 98; 2025: 63; 2026: 30.

558 papers.

Sort:

By area:

  1. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  2. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  3. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
  4. 2021A Field Guide to Federated Optimizationcited by 312
  5. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  6. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  7. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  8. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  9. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  10. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
  11. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  12. 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 112
38 more
  1. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  2. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  3. 2023Data Distillation: A Surveycited by 86
  4. 2021Federated Evaluation and Tuning for On-Device Personalization: System Design & Applicationscited by 84
  5. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 77
  6. 2021Personalized Federated Learning with Gaussian Processescited by 76
  7. 2021On Large-Cohort Training for Federated Learningcited by 73
  8. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  9. 2022An Improved Analysis of Gradient Tracking for Decentralized Machine Learningcited by 68
  10. 2021Eluding Secure Aggregation in Federated Learning via Model Inconsistencycited by 66
  11. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  12. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 60
  13. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  14. 2021Federated Multi-Armed Banditscited by 59
  15. 2021Blockchain Assisted Decentralized Federated Learning (BLADE-FL): Performance Analysis and Resource Allocationcited by 55
  16. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  17. 2021UnSplit: Data-Oblivious Model Inversion, Model Stealing, and Label Inference Attacks Against Split Learningcited by 52
  18. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 52
  19. 2022LAMP: Extracting Text from Gradients with Language Model Priorscited by 52
  20. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 50
  21. 2021Incentive Mechanisms for Federated Learning: From Economic and Game Theoretic Perspectivecited by 47
  22. 2021Opportunities of Federated Learning in Connected, Cooperative and Automated Industrial Systemscited by 45
  23. 2021Differentially Private Federated Knowledge Graphs Embeddingcited by 45
  24. 2021Survey: Leakage and Privacy at Inference Timecited by 44
  25. 2022Federated Learning via Inexact ADMMcited by 44
  26. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  27. 2021Anarchic Federated Learningcited by 42
  28. 2022A Survey on Heterogeneous Federated Learningcited by 42
  29. 2021Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiencycited by 40
  30. 2022No Free Lunch Theorem for Security and Utility in Federated Learningcited by 39
  31. 2022On-Demand Sampling: Learning Optimally from Multiple Distributionscited by 39
  32. 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
  33. 2021Deep Learning-based Patient Re-identification Is able to Exploit the Biometric Nature of Medical Chest X-ray Datacited by 38
  34. 2022Mechanisms that Incentivize Data Sharing in Federated Learningcited by 38
  35. 2022Federated Minimax Optimization: Improved Convergence Analyses and Algorithmscited by 37
  36. 2022On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyondcited by 37
  37. 2022BlindFL: Vertical Federated Machine Learning without Peeking into Your Datacited by 36
  38. 2021FedGraph: Federated Graph Learning with Intelligent Samplingcited by 34

50 most cited of 558 All 558 (JSON)