mlsanityData to 7 Oct 2026

Scaling Language Model Size in Cross-Device Federated Learning

arXiv 2204.09715 ↗First posted 31 Mar 2022cs.CL cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Most studies in cross-device federated learning focus on small models, due to the server-client communication and on-device computation bottlenecks. In this work, we leverage various techniques for mitigating these bottlenecks to train larger language models in cross-device federated learning. With systematic applications of partial model training, quantization, efficient transfer learning, and communication-efficient optimizers, we are able to train a 21M parameter Transformer and 20.2M parameter Conformer that achieve the same or better perplexity as that of a similarly sized LSTM with ~10× smaller client-to-server communication cost and 11% lower perplexity than smaller LSTMs commonly studied in literature.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

47 of 59 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2019Advances and Open Problems in Federated Learningcited by 3,021
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
35 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  5. 2020Adaptive Federated Optimizationcited by 1,036
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2018LEAF: A Benchmark for Federated Settingscited by 780
  8. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  9. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  10. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  11. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  12. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  13. 2020Rethinking Pre-training and Self-trainingcited by 352
  14. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 349
  15. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  16. 2019Revealing the Dark Secrets of BERTcited by 300
  17. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  18. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
  19. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  20. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  21. 2016Distributed Mean Estimation with Limited Communicationcited by 134
  22. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 109
  23. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  24. 2019Language Modeling with Deep Transformerscited by 96
  25. 2020Training Production Language Models without Memorizing User Datacited by 68
  26. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 68
  27. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  28. 2020Training Keyword Spotting Models on Non-IID Data with Federated Learningcited by 36
  29. 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
  30. 2019Federated Learning of N-gram Language Modelscited by 33
  31. 2021FedJAX: Federated learning simulation with JAXcited by 32
  32. 2020Pretraining Federated Text Models for Next Word Predictioncited by 22
  33. 2020Federated pretraining and fine tuning of BERT using clinical notes from multiple siloscited by 18
  34. 2021Partial Variable Training for Efficient On-Device Federated Learningcited by 14
  35. 2021Position-Invariant Truecasing with a Word-and-Character Hierarchical Recurrent Neural Networkcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 10; 2025: 4; 2026: 1.

25 papers.

Sort:

By area:

  1. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  2. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  3. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  4. 2024Federated LoRA with Sparse Communicationcited by 26
  5. 2024Asynchronous Local-SGD Training for Language Modelingcited by 24
  6. 2022Federated Select: A Primitive for Communication- and Memory-Efficient Federated Learningcited by 11
  7. 2024Trustworthy Federated Learning: Privacy, Security, and Beyondcited by 11
  8. 2024DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptationcited by 9
  9. 2022Mixed Federated Learning: Joint Decentralized and Centralized Learningcited by 8
  10. 2024No Need to Talk: Asynchronous Mixture of Language Modelscited by 5
  11. 2025FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Healthcited by 5
  12. 2024Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descentcited by 4
13 more
  1. 2022Towards Sparsified Federated Neuroimaging Models via Weight Pruningcited by 3
  2. 2025Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practicecited by 3
  3. 2023Now It Sounds Like You: Learning Personalized Vocabulary On Devicecited by 2
  4. 2023Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuningcited by 2
  5. 2023Mitigating Adversarial Attacks in Federated Learning with Trusted Execution Environmentscited by 2
  6. 2024Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translationcited by 2
  7. 2024Federated Reinforcement Learning with Constraint Heterogeneitycited by 2
  8. 2023Pelta: Shielding Transformers to Mitigate Evasion Attacks in Federated Learningcited by 1
  9. 2024Efficient Language Model Architectures for Differentially Private Federated Learningcited by 1
  10. 2024Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Securitycited by 1
  11. 2025GradualDiff-Fed: A Federated Learning Specialized Framework for Large Language Modelcited by 0
  12. 2025Scaling Law Analysis in Federated Learning: How to Select the Optimal Model Size?cited by 0
  13. 2026Differentially Private Clustered Federated Learning with Privacy-Preserving Initialization and Normality-Driven Aggregationcited by 0