mlsanityData to 14 Oct 2026

pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning

arXiv 2310.13283 ↗First posted 20 Oct 2023cs.LG cs.DCcited by 40

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is an emerging machine learning paradigm in which a central server coordinates multiple participants (clients) collaboratively to train on decentralized data. In practice, FL often faces statistical, system, and model heterogeneities, which inspires the field of Model-Heterogeneous Personalized Federated Learning (MHPFL). With the increased interest in adopting large language models (LLMs) in FL, the existing MHPFL methods cannot achieve acceptable computational and communication costs, while maintaining satisfactory model performance. To bridge this gap, we propose a novel and efficient model-heterogeneous personalized Federated learning framework based on LoRA tuning (pFedLoRA). Inspired by the popular LoRA method for fine-tuning pre-trained LLMs with a low-rank model (a.k.a., an adapter), we design a homogeneous small adapter to facilitate federated client's heterogeneous local model training with our proposed iterative training for global-local knowledge exchange. The homogeneous small local adapters are aggregated on the FL server to generate a global adapter. We theoretically prove the convergence of pFedLoRA. Extensive experiments on two benchmark datasets demonstrate that pFedLoRA outperforms six state-of-the-art baselines, beating the best method by 1.35% in test accuracy, 11.81 times computation overhead reduction and 7.41 times communication cost saving.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

28 of 44 references.

References · 28

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2016An overview of gradient descent optimization algorithmscited by 1,033
  4. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  5. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  6. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  7. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  8. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  9. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  10. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  11. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  12. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
16 more
  1. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  2. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  3. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  4. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  5. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 109
  6. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  7. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  8. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  9. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
  10. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  11. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  12. 2021Fed2: Feature-Aligned Federated Learningcited by 42
  13. 2019Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Datacited by 39
  14. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  15. 2021FedMatch: Federated Learning Over Heterogeneous Question Answering Datacited by 23
  16. 2022FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networkscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 14; 2026: 19.

40 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  3. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
  4. 2024FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learningcited by 16
  5. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 14
  6. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 12
  7. 2024Federated Continual Learning for Edge-AI: A Comprehensive Surveycited by 11
  8. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  9. 2025Mitigating Unintended Memorization with LoRA in Federated Learning for LLMscited by 7
  10. 2024SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learningcited by 5
  11. 2025ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuningcited by 4
  12. 2025FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregationcited by 3
28 more
  1. 2024Revisiting Federated Fine-Tuning: A Single Communication Round is Enough for Foundation Modelscited by 2
  2. 2025Edge-FIT: Federated Instruction Tuning of Quantized LLMs for Privacy-Preserving Smart Home Environmentscited by 2
  3. 2025FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivativecited by 2
  4. 2026Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learningcited by 2
  5. 2025Federated Multimodal Learning with Dual Adapters and Selective Pruning for Communication and Computational Efficiencycited by 1
  6. 2025FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCAcited by 1
  7. 2025FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal Large Language Modelscited by 1
  8. 2025FairFedMed: Benchmarking Group Fairness in Federated Medical Imaging with FairLoRAcited by 1
  9. 2025Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptationcited by 1
  10. 2026Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learningcited by 1
  11. 2025UnifiedFL: A Dynamic Unified Learning Framework for Equitable Federationcited by 0
  12. 2026Stabilizing Decentralized Federated Fine-Tuning via Topology-Aware Alternating LoRAcited by 0
  13. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigmcited by 0
  14. 2026FedGRPO: Privately Optimizing Foundation Models with Group-Relative Rewards from Domain Clientcited by 0
  15. 2026SecureGate: Learning When to Reveal PII Safely via Token-Gated Dual-Adapters for Federated LLMscited by 0
  16. 2026HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learningcited by 0
  17. 2026FedSDR: Federated Self-Distillation with Rectificationcited by 0
  18. 2026Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Modelscited by 0
  19. 2026Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentationcited by 0
  20. 2026PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMscited by 0
  21. 2026Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learningcited by 0
  22. 2026FedLSG: LLM-Enhanced Semantic Calibration for Federated Graph Backdoor Defensecited by 0
  23. 2026EFFEKT: Efficient Federated Knowledge Transfer to Foundation Modelscited by 0
  24. 2026UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneitycited by 0
  25. 2026Co-VLA: Consensus-based Federated Training for Vision-Language-Action Modelscited by 0
  26. 2026Perplexity Predicts Protection: Choosing Pretrained Backbones for Worst-Client Fairness in Federated Parameter-Efficient Fine-Tuningcited by 0
  27. 2026FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projectioncited by 0
  28. 2026Personalized Federated Direct Preference Optimization via LoRA Adapterscited by 0