# Mixed Federated Learning: Joint Decentralized and Centralized Learning

arXiv: https://arxiv.org/abs/2205.13655 · HTML: https://mlsanity.com/paper/2205.13655/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 26 May 2022.

## Builds on (most cited first, 29 of 29)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Distributed learning of deep neural network over multiple agents](https://mlsanity.com/paper/1810.06060/) (2018)
- [SCAFFOLD: Stochastic Controlled Averaging for Federated Learning](https://mlsanity.com/paper/1910.06378/) (2019)
- [Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning](https://mlsanity.com/paper/2008.03606/) (2020)
- [Federated Learning for Emoji Prediction in a Mobile Keyboard](https://mlsanity.com/paper/1906.04329/) (2019)
- [Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification](https://mlsanity.com/paper/2007.03813/) (2020)
- [Unified Optimal Analysis of the (Stochastic) Gradient Method](https://mlsanity.com/paper/1907.04232/) (2019)
- [Federated Reconstruction: Partially Local Federated Learning](https://mlsanity.com/paper/2102.03448/) (2021)
- [Training Production Language Models without Memorizing User Data](https://mlsanity.com/paper/2009.10031/) (2020)
- [Public Data-Assisted Mirror Descent for Private Model Training](https://mlsanity.com/paper/2112.00193/) (2021)
- [Private Adaptive Gradient Methods for Convex Optimization](https://mlsanity.com/paper/2106.13756/) (2021)
- [Private Adaptive Optimization with Side Information](https://mlsanity.com/paper/2202.05963/) (2022)
- [Learning Spread-out Local Feature Descriptors](https://mlsanity.com/paper/1708.06320/) (2017)
- [Scaling Language Model Size in Cross-Device Federated Learning](https://mlsanity.com/paper/2204.09715/) (2022)
- [Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory](https://mlsanity.com/paper/2201.02664/) (2022)
- [Production federated keyword spotting via distillation, filtering, and joint federated-centralized training](https://mlsanity.com/paper/2204.06322/) (2022)
- [Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shift](https://mlsanity.com/paper/2111.12150/) (2021)
- [Learning Federated Representations and Recommendations with Limited Negatives](https://mlsanity.com/paper/2108.07931/) (2021)

## Built on by (most cited first, 8 of 8)

- [GPT-FL: Generative Pre-trained Model-Assisted Federated Learning](https://mlsanity.com/paper/2306.02210/) (2023)
- [Accelerating Hybrid Federated Learning Convergence under Partial Participation](https://mlsanity.com/paper/2304.05397/) (2023)
- [The Federation Strikes Back: A Survey of Federated Learning Privacy Attacks, Defenses, Applications, and Policy Landscape](https://mlsanity.com/paper/2405.03636/) (2024)
- [Federated Learning with Server Learning: Enhancing Performance for Non-IID Data](https://mlsanity.com/paper/2210.02614/) (2022)
- [Federated Training of Dual Encoding Models on Small Non-IID Client Datasets](https://mlsanity.com/paper/2210.00092/) (2022)
- [Towards Federated Learning Under Resource Constraints via Layer-wise Training and Depth Dropout](https://mlsanity.com/paper/2309.05213/) (2023)
- [The Gift of Feedback: Improving ASR Model Quality by Learning from User Corrections through Federated Learning](https://mlsanity.com/paper/2310.00141/) (2023)
- [Learning from straggler clients in federated learning](https://mlsanity.com/paper/2403.09086/) (2024)

## Abstract (arXiv)

Federated learning (FL) enables learning from decentralized privacy-sensitive data, with computations on raw data confined to take place at edge clients. This paper introduces mixed FL, which incorporates an additional loss term calculated at the coordinating server (while maintaining FL's private data restrictions). There are numerous benefits. For example, additional datacenter data can be leveraged to jointly learn from centralized (datacenter) and decentralized (federated) training data and better match an expected inference data distribution. Mixed FL also enables offloading some intensive computations (e.g., embedding regularization) to the server, greatly reducing communication and client computation load. For these and other mixed FL use cases, we present three algorithms: PARALLEL TRAINING, 1-WAY GRADIENT TRANSFER, and 2-WAY GRADIENT TRANSFER. We state convergence bounds for each, and give intuition on which are suited to particular mixed FL problems. Finally we perform extensive experiments on three tasks, demonstrating that mixed FL can blend training data to achieve an oracle's accuracy on an inference distribution, and can reduce communication and computation overhead by over 90%. Our experiments confirm theoretical predictions of how algorithms perform under different mixed FL problem settings.