# Federated Optimization:Distributed Optimization Beyond the Datacenter

arXiv: https://arxiv.org/abs/1511.03575 · HTML: https://mlsanity.com/paper/1511.03575/

## Key facts

- Cited by 195 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2015.

## Built on by (most cited first, 30 of 195)

- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Models](https://mlsanity.com/paper/2110.13057/) (2021)
- [Fast Federated Learning in the Presence of Arbitrary Device Unavailability](https://mlsanity.com/paper/2106.04159/) (2021)
- [TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation](https://mlsanity.com/paper/2303.06937/) (2023)
- [Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances](https://mlsanity.com/paper/2111.00418/) (2021)
- [Federated Learning via Inexact ADMM](https://mlsanity.com/paper/2204.10607/) (2022)
- [FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learning](https://mlsanity.com/paper/2103.13822/) (2021)
- [Gradient Disaggregation: Breaking Privacy in Federated Learning by Reconstructing the User Participant Matrix](https://mlsanity.com/paper/2106.06089/) (2021)
- [Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective](https://mlsanity.com/paper/2111.03741/) (2021)
- [Scaling Federated Learning for Fine-tuning of Large Language Models](https://mlsanity.com/paper/2102.00875/) (2021)
- [Non-IID data and Continual Learning processes in Federated Learning: A long road ahead](https://mlsanity.com/paper/2111.13394/) (2021)
- [Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization](https://mlsanity.com/paper/2203.09747/) (2022)
- [Concept drift detection and adaptation for federated and continual learning](https://mlsanity.com/paper/2105.13309/) (2021)
- [Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning](https://mlsanity.com/paper/2102.03198/) (2021)
- [Ground-Assisted Federated Learning in LEO Satellite Constellations](https://mlsanity.com/paper/2109.01348/) (2021)
- [SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks](https://mlsanity.com/paper/2106.02743/) (2021)
- [FEDZIP: A Compression Framework for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2102.01593/) (2021)
- [FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulations](https://mlsanity.com/paper/2203.13789/) (2022)
- [Reconstructing Training Data from Model Gradient, Provably](https://mlsanity.com/paper/2212.03714/) (2022)
- [DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2309.13546/) (2023)
- [GradientCoin: A Peer-to-Peer Decentralized Large Language Models](https://mlsanity.com/paper/2308.10502/) (2023)
- [Federated Learning with Taskonomy for Non-IID Data](https://mlsanity.com/paper/2103.15947/) (2021)
- [Federated Learning with Heterogeneous Architectures using Graph HyperNetworks](https://mlsanity.com/paper/2201.08459/) (2022)
- [Communication-Efficient Federated Learning via Quantized Compressed Sensing](https://mlsanity.com/paper/2111.15071/) (2021)
- [Communication-Efficient Split Learning via Adaptive Feature-Wise Compression](https://mlsanity.com/paper/2307.10805/) (2023)
- [FedHCA2: Towards Hetero-Client Federated Multi-Task Learning](https://mlsanity.com/paper/2311.13250/) (2023)
- [A Decentralized Federated Learning Framework via Committee Mechanism with Convergence Guarantee](https://mlsanity.com/paper/2108.00365/) (2021)
- [KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey](https://mlsanity.com/paper/2110.01858/) (2021)
- [An Equivalence Between Data Poisoning and Byzantine Gradient Attacks](https://mlsanity.com/paper/2202.08578/) (2022)
- [Defending against the Label-flipping Attack in Federated Learning](https://mlsanity.com/paper/2207.01982/) (2022)

## Abstract (arXiv)

We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are distributed (unevenly) over an extremely large number of nodes, but the goal remains to train a high-quality centralized model. We refer to this setting as Federated Optimization. In this setting, communication efficiency is of utmost importance. A motivating example for federated optimization arises when we keep the training data locally on users' mobile devices rather than logging it to a data center for training. Instead, the mobile devices are used as nodes performing computation on their local data in order to update a global model. We suppose that we have an extremely large number of devices in our network, each of which has only a tiny fraction of data available totally; in particular, we expect the number of data points available locally to be much smaller than the number of devices. Additionally, since different users generate data with different patterns, we assume that no device has a representative sample of the overall distribution. We show that existing algorithms are not suitable for this setting, and propose a new algorithm which shows encouraging experimental results. This work also sets a path for future research needed in the context of federated optimization.