mlsanityData to 7 Oct 2026

When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods

arXiv 2212.10025 ↗First posted 20 Dec 2022cs.LG cs.CLcited by 74

Not on the map. Why?

Abstract (arXiv)

With increasing privacy concerns on data, recent studies have made significant progress using federated learning (FL) on privacy-sensitive natural language processing (NLP) tasks. Much literature suggests fully fine-tuning pre-trained language models (PLMs) in the FL paradigm can mitigate the data heterogeneity problem and close the performance gap with centralized training. However, large PLMs bring the curse of prohibitive communication overhead and local model adaptation costs for the FL system. To this end, we introduce various parameter-efficient tuning (PETuning) methods into federated learning. Specifically, we provide a holistic empirical study of representative PLMs tuning methods in FL. The experimental results cover the analysis of data heterogeneity levels, data scales, and different FL scenarios. Overall communication overhead can be significantly reduced by locally tuning and globally aggregating lightweight model parameters while maintaining acceptable performance in various FL settings. To facilitate the research of PETuning in FL, we also develop a federated tuning framework FedPETuning, which allows practitioners to exploit different PETuning methods under the FL training paradigm conveniently. The source code is available at urlhttps://github.com/iezhuozhuo/FedETuning/tree/deltaTuning.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

33 of 45 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2019Advances and Open Problems in Federated Learningcited by 3,021
  11. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
21 more
  1. 2018Federated Learning with Non-IID Datacited by 1,099
  2. 2019Deep Leakage from Gradientscited by 1,004
  3. 2019Towards Federated Learning at Scale: System Designcited by 826
  4. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  5. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  6. 2020Information Leakage in Embedding Modelscited by 199
  7. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  8. 2021Federated Learning for Open Bankingcited by 138
  9. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  10. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  11. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
  12. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  13. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  14. 2021Natural Language Understanding with Privacy-Preserving BERTcited by 54
  15. 2021FedLab: A Flexible Federated Learning Frameworkcited by 54
  16. 2018Learning Private Neural Language Modeling with Attentive Aggregationcited by 49
  17. 2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 36
  18. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  19. 2022Pretrained Models for Multilingual Federated Learningcited by 30
  20. 2021Privacy enabled Financial Text Classification using Differential Privacy and Federated Learningcited by 18
  21. 2022Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devicescited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 32; 2025: 27; 2026: 8.

74 papers.

Sort:

By area:

  1. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  2. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  3. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
  4. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 54
  5. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  6. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  7. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  8. 2024PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restorationcited by 27
  9. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  10. 2024Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecastingcited by 26
  11. 2024Federated LoRA with Sparse Communicationcited by 26
  12. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
38 more
  1. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  2. 2024Federated Large Language Models: Current Progress and Future Directionscited by 22
  3. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
  4. 2023Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Uglycited by 17
  5. 2024FedPFT: Federated Proxy Fine-Tuning of Foundation Modelscited by 16
  6. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
  7. 2024Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Modelscited by 15
  8. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  9. 2024AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guaranteescited by 14
  10. 2024Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Modelscited by 12
  11. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 12
  12. 2024AFL: A Single-Round Analytic Approach for Federated Learning with Pre-trained Modelscited by 11
  13. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
  14. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  15. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  16. 2025Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuningcited by 7
  17. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  18. 2024Fine-Tuning and Deploying Large Language Models Over Edges: Issues and Approachescited by 6
  19. 2024Ferret: Federated Full-Parameter Tuning at Scale for Large Language Modelscited by 6
  20. 2023FedJudge: Federated Legal Large Language Modelcited by 5
  21. 2024Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learningcited by 5
  22. 2024Federated Co-tuning Framework for Large and Small Language Modelscited by 5
  23. 2024Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devicescited by 5
  24. 2024DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptationcited by 4
  25. 2025Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Datacited by 4
  26. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  27. 2024MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Modelscited by 3
  28. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  29. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Modelscited by 3
  30. 2023FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computingcited by 2
  31. 2024EFTViT: Efficient Federated Training of Vision Transformers with Masked Images on Resource-Constrained Clientscited by 2
  32. 2025Rate-My-LoRA: Efficient and Adaptive Federated Model Tuning for Cardiac MRI Segmentationcited by 2
  33. 2025Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptationcited by 2
  34. 2025Federated Adapter on Foundation Models: An Out-Of-Distribution Approachcited by 2
  35. 2026Toward Enhancing Representation Learning in Federated Multi-Task Settingscited by 2
  36. 2023Orchestration of Emulator Assisted Mobile Edge Tuning for AI Foundation Models: A Multi-Agent Deep Reinforcement Learning Approachcited by 1
  37. 2024The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustnesscited by 1
  38. 2024Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Securitycited by 1

50 most cited of 74 All 74 (JSON)