mlsanityData to 14 Oct 2026

FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA

arXiv 2503.11880 ↗First posted 14 Mar 2025cs.LG cs.AI cs.CLcited by 13

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

Fine-tuning large language models (LLMs) in federated settings enables privacy-preserving adaptation but suffers from cross-client interference due to model aggregation. Existing federated LoRA fine-tuning methods, primarily based on FedAvg, struggle with data heterogeneity, leading to harmful cross-client interference and suboptimal personalization. In this work, we propose FedALT, a novel personalized federated LoRA fine-tuning algorithm that fundamentally departs from FedAvg. Instead of using an aggregated model to initialize local training, each client continues training its individual LoRA while incorporating shared knowledge through a separate Rest-of-World (RoW) LoRA component. To effectively balance local adaptation and global information, FedALT introduces an adaptive mixer that dynamically learns input-specific weightings between the individual and RoW LoRA components, drawing conceptual foundations from the Mixture-of-Experts (MoE) paradigm. Through extensive experiments on NLP benchmarks, we demonstrate that FedALT significantly outperforms state-of-the-art personalized federated LoRA fine-tuning methods, achieving superior local adaptation without sacrificing computational efficiency.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 45 of its 53 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

45 of 53 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
33 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  4. 2024Large Language Models: A Surveycited by 525
  5. 2021Towards Personalized Federated Learningcited by 464
  6. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  7. 2020Adaptive Personalized Federated Learningcited by 358
  8. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  9. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
  10. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  11. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  12. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  13. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  14. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  15. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  16. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  17. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  18. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  19. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  20. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  21. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  22. 2024Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Datacited by 66
  23. 2024FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuningcited by 59
  24. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  25. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  26. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 38
  27. 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
  28. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  29. 2024Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domainscited by 28
  30. 2023Accelerating Hybrid Federated Learning Convergence under Partial Participationcited by 15
  31. 2023Federated Learning with Instance-Dependent Noisy Labelcited by 9
  32. 2022A Benchmark for Federated Hetero-Task Learningcited by 6
  33. 2025Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRAcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  2. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
  3. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  4. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 5
  5. 2025Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAscited by 3
  6. 2026FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuningcited by 2
  7. 2025FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCAcited by 1
  8. 2026Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneitycited by 1
  9. 2026RW-LoRA: Communication-Efficient Decentralized LoRA Fine-Tuning via Random Walkscited by 1
  10. 2025Computation-aware Energy-harvesting Federated Learning with Pipelined Cyclic Schedulingcited by 0
  11. 2026HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learningcited by 0
  12. 2026FedLSG: LLM-Enhanced Semantic Calibration for Federated Graph Backdoor Defensecited by 0
1 more
  1. 2026CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
  2. 2026FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks
  3. 2025FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning
  4. 2024HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantization
  5. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
  6. 2026FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization
  7. 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
  8. 2025Decentralized Low-Rank Fine-Tuning of Large Language Models