mlsanityData to 7 Oct 2026

FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks

arXiv 2104.08815 ↗First posted 18 Apr 2021cs.CL cs.AI cs.LGcited by 84

Not on the map. Why?

Abstract (arXiv)

Increasing concerns and regulations about data privacy and sparsity necessitate the study of privacy-preserving, decentralized learning methods for natural language processing (NLP) tasks. Federated learning (FL) provides promising approaches for a large number of clients (e.g., personal devices or organizations) to collaboratively learn a shared global model to benefit all clients while allowing users to keep their data locally. Despite interest in studying FL methods for NLP tasks, a systematic comparison and analysis is lacking in the literature. Herein, we present the FedNLP, a benchmarking framework for evaluating federated learning methods on four different task formulations: text classification, sequence tagging, question answering, and seq2seq. We propose a universal interface between Transformer-based language models (e.g., BERT, BART) and FL methods (e.g., FedAvg, FedOPT, etc.) under various non-IID partitioning strategies. Our extensive experiments with FedNLP provide empirical comparisons between FL methods and helps us better understand the inherent challenges of this direction. The comprehensive analysis points to intriguing and exciting future research aimed at developing FL methods for NLP tasks.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 46 of its 71 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

46 of 71 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2019Advances and Open Problems in Federated Learningcited by 3,021
  12. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
34 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  3. 2020Adaptive Federated Optimizationcited by 1,036
  4. 2019Deep Leakage from Gradientscited by 1,004
  5. 2018LEAF: A Benchmark for Federated Settingscited by 780
  6. 2017Federated Multi-Task Learningcited by 669
  7. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  8. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  9. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  10. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  11. 2020Federated Learning with Matched Averagingcited by 564
  12. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  13. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  14. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  15. 2020Attack of the Tails: Yes, You Really Can Backdoor Federated Learningcited by 307
  16. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  17. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  18. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  19. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  20. 2020Privacy and Robustness in Federated Learning: Attacks and Defensescited by 163
  21. 2019Federated Learning for Emoji Prediction in a Mobile Keyboardcited by 115
  22. 2018A generic framework for privacy preserving deep learningcited by 103
  23. 2018Federated Learning for Keyword Spottingcited by 95
  24. 2020Byzantine-Resilient Secure Federated Learningcited by 93
  25. 2020Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learningcited by 78
  26. 2020MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulationcited by 56
  27. 2018Learning Private Neural Language Modeling with Attentive Aggregationcited by 49
  28. 2019Federated Learning of N-gram Language Modelscited by 33
  29. 2019Central Server Free Federated Learning over Single-sided Trust Social Networkscited by 33
  30. 2020Coded Computing for Low-Latency Federated Learning over Wireless Edge Networkscited by 32
  31. 2020FedNER: Privacy-preserving Medical Named Entity Recognition with Federated Learningcited by 24
  32. 2020Pretraining Federated Text Models for Next Word Predictioncited by 22
  33. 2019CodedPrivateML: A Fast and Privacy-Preserving Framework for Distributed Machine Learningcited by 20
  34. 2020Federated pretraining and fine tuning of BERT using clinical notes from multiple siloscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 15; 2023: 18; 2024: 20; 2025: 22; 2026: 7.

84 papers.

Sort:

By area:

  1. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  2. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  3. 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 121
  4. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  5. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  6. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  7. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  8. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  9. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  10. 2022Federated Learning for Medical Applications: A Taxonomy, Current Trends, Challenges, and Future Research Directionscited by 34
  11. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  12. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
38 more
  1. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  2. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  3. 2023Federated Domain Generalization: A Surveycited by 25
  4. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
  5. 2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 24
  6. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  7. 2022FLAIR: Federated Learning Annotated Image Repositorycited by 22
  8. 2021FedDAG: Federated DAG Structure Learningcited by 17
  9. 2021Collaborative Semantic Aggregation and Calibration for Federated Domain Generalizationcited by 16
  10. 2022Decentral and Incentivized Federated Learning Frameworks: A Systematic Literature Reviewcited by 15
  11. 2022Federated Few-Shot Learning for Mobile NLPcited by 15
  12. 2023On the Efficacy of Differentially Private Few-shot Image Classificationcited by 12
  13. 2023FedTherapist: Mental Health Monitoring with User-Generated Linguistic Expressions on Smartphones via Federated Learningcited by 10
  14. 2024Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneitycited by 10
  15. 2024LoRA-LiteE: A Computationally Efficient Framework for Chatbot Preference-Tuningcited by 10
  16. 2023FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Thingscited by 8
  17. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  18. 2024MAP: Model Aggregation and Personalization in Federated Learning with Incomplete Classescited by 7
  19. 2024COALA: A Practical and Vision-Centric Federated Learning Platformcited by 7
  20. 2025Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuningcited by 7
  21. 2023Knowledge-Enhanced Semi-Supervised Federated Learning for Aggregating Heterogeneous Lightweight Clients in IoTcited by 6
  22. 2023A Privacy Preserving System for Movie Recommendations Using Federated Learningcited by 6
  23. 2023Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarizationcited by 6
  24. 2022Towards Practical Few-shot Federated NLPcited by 5
  25. 2022Social-Aware Clustered Federated Learning with Customized Privacy Preservationcited by 5
  26. 2023CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Modelcited by 5
  27. 2024Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devicescited by 5
  28. 2025Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languagescited by 5
  29. 2025FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Healthcited by 5
  30. 2023Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithmscited by 4
  31. 2025Large Multimodal Models for Low-Resource Languages: A Surveycited by 4
  32. 2025Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Datacited by 4
  33. 2024PPFedIT: Towards Privacy-Preserving Federated Instruction Tuning with Few-shot Local Examplescited by 3
  34. 2024MedLeak: Multimodal Medical Data Leakage in Secure Federated Learning with Crafted Modelscited by 3
  35. 2026Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluationscited by 3
  36. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  37. 2023Now It Sounds Like You: Learning Personalized Vocabulary On Devicecited by 2
  38. 2024Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translationcited by 2

50 most cited of 84 All 84 (JSON)