mlsanityData to 14 Oct 2026

LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement

arXiv 2411.14961 ↗First posted 22 Nov 2024cs.LG cs.CVcited by 38

Not on the map. Why?

Abstract (arXiv)

Foundation models (FMs) achieve strong performance across diverse tasks with task-specific fine-tuning, yet full parameter fine-tuning is often computationally prohibitive for large models. Parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) reduce this cost by introducing low-rank matrices for tuning fewer parameters. While LoRA allows for efficient fine-tuning, it requires significant data for adaptation, making Federated Learning (FL) an appealing solution due to its privacy-preserving collaborative framework. However, combining LoRA with FL introduces two key challenges: the Server-Side Aggregation Bias, where server-side averaging of LoRA matrices diverges from the ideal global update, and the Client-Side Initialization Lag, emphasizing the need for consistent initialization across rounds. Existing approaches address these challenges individually, limiting their effectiveness. We propose LoRA-FAIR, a novel method that tackles both issues by introducing a correction term on the server, enhancing aggregation efficiency and accuracy. LoRA-FAIR maintains computational and communication efficiency, yielding superior performance over state-of-the-art methods. Experimental results on ViT and MLP-Mixer models across large-scale datasets demonstrate that LoRA-FAIR consistently achieves performance improvements in FL settings.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

39 of 51 references.

References · 39

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  11. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  12. 2018Federated Learning with Non-IID Datacited by 1,108
27 more
  1. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  3. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  4. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  5. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  6. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  7. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  8. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  9. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  10. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  11. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 142
  12. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  13. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  14. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  15. 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 96
  16. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  17. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
  18. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  19. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  20. 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 36
  21. 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
  22. 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 30
  23. 2024Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domainscited by 28
  24. 2023FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clientscited by 19
  25. 2024FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathologycited by 17
  26. 2023Federated Learning with Instance-Dependent Noisy Labelcited by 9
  27. 2024Prioritizing Modalities: Flexible Importance Scheduling in Federated Multimodal Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 24.

38 papers.

Sort:

By area:

  1. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  2. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
  3. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
  4. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
  5. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  6. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  7. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuningcited by 6
  8. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 5
  9. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 3
  10. 2025Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuningcited by 2
  11. 2026Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignmentcited by 2
  12. 2026FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuningcited by 2
26 more
  1. 2026Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRAcited by 2
  2. 2024Sequential Compression Layers for Efficient Federated Learning in Foundational Modelscited by 1
  3. 2025FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraintscited by 1
  4. 2026SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clientscited by 1
  5. 2026Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMscited by 1
  6. 2026Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformerscited by 1
  7. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  8. 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRAcited by 1
  9. 2025Frugal Federated Learning for Violence Detection: A Comparison of LoRA-Tuned VLMs and Personalized CNNscited by 0
  10. 2025Federated Learning for Video Violence Detection: Complementary Roles of Lightweight CNNs and Vision-Language Models for Energy-Efficient Usecited by 0
  11. 2025ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregationcited by 0
  12. 2026FedDetox: Robust Federated SLM Alignment via On-Device Data Sanitizationcited by 0
  13. 2026Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMscited by 0
  14. 2026FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptationcited by 0
  15. 2026Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Modelscited by 0
  16. 2026Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentationcited by 0
  17. 2026Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuningcited by 0
  18. 2026MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuningcited by 0
  19. 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approachcited by 0
  20. 2026Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneitycited by 0
  21. 2026What Should Federated LoRA Share? FedSAIL via Input-aware Subspace Alignmentcited by 0
  22. 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregationcited by 0
  23. 2026Federated Agent Optimizationcited by 0
  24. 2026FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantizationcited by 0
  25. 2026FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projectioncited by 0
  26. 2026Personalized Federated Direct Preference Optimization via LoRA Adapterscited by 0