mlsanityData to 14 Oct 2026

Federated Select: A Primitive for Communication- and Memory-Efficient Federated Learning

arXiv 2208.09432 ↗First posted 19 Aug 2022cs.LG cs.DCcited by 11

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a framework for machine learning across heterogeneous client devices in a privacy-preserving fashion. To date, most FL algorithms learn a "global" server model across multiple rounds. At each round, the same server model is broadcast to all participating clients, updated locally, and then aggregated across clients. In this work, we propose a more general procedure in which clients "select" what values are sent to them. Notably, this allows clients to operate on smaller, data-dependent slices. In order to make this practical, we outline a primitive, federated select, which enables client-specific selection in realistic FL systems. We discuss how to use federated select for model training and show that it can lead to drastic reductions in communication and client memory usage, potentially enabling the training of models too large to fit on-device. We also discuss the implications of federated select on privacy and trust, which in turn affect possible system constraints and design. Finally, we discuss open questions concerning model architectures, privacy-preserving technologies, and practical FL systems.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 19 Aug 2022.

Builds on

29 of 40 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  9. 2020Adaptive Federated Optimizationcited by 1,046
  10. 2019Towards Federated Learning at Scale: System Designcited by 832
  11. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  12. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
17 more
  1. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  2. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  3. 2021A Field Guide to Federated Optimizationcited by 315
  4. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  5. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  6. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
  7. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  8. 2019Language Modeling with Deep Transformerscited by 96
  9. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 90
  10. 2021On Large-Cohort Training for Federated Learningcited by 74
  11. 2019Federated Heavy Hitters Discovery with Differential Privacycited by 32
  12. 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
  13. 2020FedSKETCH: Communication-Efficient and Private Federated Learning via Sketchingcited by 20
  14. 2021Partial Variable Training for Efficient On-Device Federated Learningcited by 14
  15. 2022Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theorycited by 14
  16. 2021Towards Sparse Federated Analytics: Location Heatmaps under Distributed Differential Privacy with Secure Aggregationcited by 10
  17. 2022Efficient Image Representation Learning with Federated Sampled Softmaxcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 3; 2025: 0; 2026: 1.

11 papers.

  1. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  2. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  3. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  4. 2024Federated LoRA with Sparse Communicationcited by 26
  5. 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
  6. 2024DrJAX: Scalable and Differentiable MapReduce Primitives in JAXcited by 9
  7. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  8. 2023Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beatcited by 4
  9. 2023Federated Automatic Differentiationcited by 2
  10. 2026FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projectioncited by 1
  11. 2023Tackling Computational Heterogeneity in FL: A Few Theoretical Insightscited by 0