mlsanityData to 7 Oct 2026

Pretrained Models for Multilingual Federated Learning

arXiv 2206.02291 ↗First posted 6 Jun 2022cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Since the advent of Federated Learning (FL), research has applied these methods to natural language processing (NLP) tasks. Despite a plethora of papers in FL for NLP, no previous works have studied how multilingual text impacts FL algorithms. Furthermore, multilingual text provides an interesting avenue to examine the impact of non-IID text (e.g. different languages) on FL in naturally occurring data. We explore three multilingual language tasks, language modeling, machine translation, and text classification using differing federated and non-federated learning algorithms. Our results show that using pretrained models reduces the negative effects of FL, helping them to perform near or better than centralized (no privacy) learning, even when using non-IID partitioning.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

30 of 46 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  11. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  12. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
18 more
  1. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  2. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  3. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  4. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  5. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  6. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  7. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  8. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  9. 2019Federated Learning for Emoji Prediction in a Mobile Keyboardcited by 115
  10. 2018MTNT: A Testbed for Machine Translation of Noisy Textcited by 54
  11. 2019Federated Learning Of Out-Of-Vocabulary Wordscited by 51
  12. 2018Learning Private Neural Language Modeling with Attentive Aggregationcited by 49
  13. 2021Federated Learning Meets Natural Language Processing: A Surveycited by 42
  14. 2019Federated Learning of N-gram Language Modelscited by 33
  15. 2020Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translationcited by 28
  16. 2020Pretraining Federated Text Models for Next Word Predictioncited by 22
  17. 2020Federated pretraining and fine tuning of BERT using clinical notes from multiple siloscited by 18
  18. 2019Findings of the First Shared Task on Machine Translation Robustnesscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 10; 2024: 11; 2025: 2; 2026: 3.

30 papers.

Sort:

By area:

  1. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  2. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  3. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  4. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  5. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  6. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  7. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  8. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  9. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  10. 2023FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Modelscited by 25
  11. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  12. 2023On the Efficacy of Differentially Private Few-shot Image Classificationcited by 12
18 more
  1. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  2. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  3. 2023FedYolo: Augmenting Federated Learning with Pretrained Transformerscited by 9
  4. 2023Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuningcited by 9
  5. 2023Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adaptercited by 8
  6. 2024C2A: Client-Customized Adaptation for Parameter-Efficient Federated Learningcited by 8
  7. 2025Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languagescited by 5
  8. 2023Federated Learning Based Multilingual Emoji Prediction In Clean and Attack Scenarioscited by 3
  9. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  10. 2024Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translationcited by 2
  11. 2023Personalized Federated Learning via Gradient Modulation for Heterogeneous Text Summarizationcited by 1
  12. 2023Fingerprint Attack: Client De-Anonymization in Federated Learningcited by 1
  13. 2024Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Securitycited by 1
  14. 2024A Survey for Large Language Models in Biomedicinecited by 1
  15. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  16. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 0
  17. 2026Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Compositioncited by 0
  18. 2026DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Datacited by 0