mlsanityData to 7 Oct 2026

Mixed Federated Learning: Joint Decentralized and Centralized Learning

arXiv 2205.13655 ↗First posted 26 May 2022cs.LG cs.DCcited by 8

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) enables learning from decentralized privacy-sensitive data, with computations on raw data confined to take place at edge clients. This paper introduces mixed FL, which incorporates an additional loss term calculated at the coordinating server (while maintaining FL's private data restrictions). There are numerous benefits. For example, additional datacenter data can be leveraged to jointly learn from centralized (datacenter) and decentralized (federated) training data and better match an expected inference data distribution. Mixed FL also enables offloading some intensive computations (e.g., embedding regularization) to the server, greatly reducing communication and client computation load. For these and other mixed FL use cases, we present three algorithms: PARALLEL TRAINING, 1-WAY GRADIENT TRANSFER, and 2-WAY GRADIENT TRANSFER. We state convergence bounds for each, and give intuition on which are suited to particular mixed FL problems. Finally we perform extensive experiments on three tasks, demonstrating that mixed FL can blend training data to achieve an oracle's accuracy on an inference distribution, and can reduce communication and computation overhead by over 90%. Our experiments confirm theoretical predictions of how algorithms perform under different mixed FL problem settings.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

29 of 42 references.

References · 29

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2018Federated Learning with Non-IID Datacited by 1,099
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2018LEAF: A Benchmark for Federated Settingscited by 780
  9. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  10. 2019Federated Learning with Personalization Layerscited by 533
  11. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 391
  12. 2021A Field Guide to Federated Optimizationcited by 312
17 more
  1. 2018Distributed learning of deep neural network over multiple agentscited by 277
  2. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  3. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  4. 2019Federated Learning for Emoji Prediction in a Mobile Keyboardcited by 115
  5. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  6. 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 81
  7. 2021Federated Reconstruction: Partially Local Federated Learningcited by 78
  8. 2020Training Production Language Models without Memorizing User Datacited by 68
  9. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  10. 2021Private Adaptive Gradient Methods for Convex Optimizationcited by 46
  11. 2022Private Adaptive Optimization with Side Informationcited by 38
  12. 2017Learning Spread-out Local Feature Descriptorscited by 34
  13. 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
  14. 2022Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theorycited by 14
  15. 2022Production federated keyword spotting via distillation, filtering, and joint federated-centralized trainingcited by 9
  16. 2021Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shiftcited by 7
  17. 2021Learning Federated Representations and Recommendations with Limited Negativescited by 6

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4; 2024: 2.

8 papers.

  1. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  2. 2023Accelerating Hybrid Federated Learning Convergence under Partial Participationcited by 15
  3. 2024The Federation Strikes Back: A Survey of Federated Learning Privacy Attacks, Defenses, Applications, and Policy Landscapecited by 13
  4. 2022Federated Learning with Server Learning: Enhancing Performance for Non-IID Datacited by 6
  5. 2022Federated Training of Dual Encoding Models on Small Non-IID Client Datasetscited by 2
  6. 2023Towards Federated Learning Under Resource Constraints via Layer-wise Training and Depth Dropoutcited by 2
  7. 2023The Gift of Feedback: Improving ASR Model Quality by Learning from User Corrections through Federated Learningcited by 2
  8. 2024Learning from straggler clients in federated learningcited by 1