mlsanityData to 7 Oct 2026

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources

arXiv 2402.11505 ↗First posted 18 Feb 2024cs.CL cs.AIcited by 93

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) has recently been applied to the parameter-efficient fine-tuning of Large Language Models (LLMs). While promising, it raises significant challenges due to the heterogeneous resources and data distributions of clients. This study introduces FlexLoRA, a simple yet effective aggregation scheme for LLM fine-tuning, which mitigates the ``bucket effect'' in traditional FL that restricts the potential of clients with ample resources by tying them to the capabilities of the least-resourced participants. FlexLoRA allows for dynamic adjustment of local LoRA ranks, fostering the development of a global model imbued with broader, less task-specific knowledge. By synthesizing a full-size LoRA weight from individual client contributions and employing Singular Value Decomposition (SVD) for weight redistribution, FlexLoRA fully leverages heterogeneous client resources. Involving thousands of clients performing heterogeneous NLP tasks and client resources, our experiments validate the efficacy of FlexLoRA, with the federated global model achieving consistently better improvement over SOTA FL methods in downstream NLP task performance across various heterogeneous distributions. FlexLoRA's practicality is further underscored by our theoretical analysis and its seamless integration with existing LoRA-based FL methods, offering a path toward cross-device, privacy-preserving federated tuning for LLMs.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

36 of 45 references.

References · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  10. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
24 more
  1. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  2. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  3. 2017Learning multiple visual domains with residual adapterscited by 553
  4. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  5. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  6. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  7. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  8. 2021Personalized Federated Learning using Hypernetworkscited by 210
  9. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  10. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  11. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  12. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  13. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  14. 2019Variational Federated Multi-Task Learningcited by 69
  15. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  16. 2023Data-Juicer: A One-Stop Data Processing System for Large Language Modelscited by 52
  17. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  18. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  19. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  20. 2023FS-Real: Towards Real-World Cross-Device Federated Learningcited by 22
  21. 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 18
  22. 2022Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devicescited by 16
  23. 2022Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularizationcited by 12
  24. 2023A Stability Analysis of Fine-Tuning a Pre-Trained Modelcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 29; 2026: 48.

93 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  3. 2024Federated LoRA with Sparse Communicationcited by 26
  4. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  5. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  6. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
  7. 2024Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Modelscited by 12
  8. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  9. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
  10. 2024Federated Data-Efficient Instruction Tuning for Large Language Modelscited by 9
  11. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  12. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
38 more
  1. 2025Federated Fine-Tuning of LLMs: Framework Comparison and Research Directionscited by 8
  2. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  3. 2026FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRAcited by 8
  4. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  5. 2024On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialistscited by 6
  6. 2024Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Modelscited by 6
  7. 2024SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learningcited by 5
  8. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  9. 2025FedSRD: Sparsify-Reconstruct-Decompose for Communication-Efficient Federated Large Language Models Fine-Tuningcited by 5
  10. 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignmentcited by 5
  11. 2025Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Modelscited by 4
  12. 2025Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Datacited by 4
  13. 2025FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Modelscited by 4
  14. 2025An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Datacited by 4
  15. 2025ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuningcited by 4
  16. 2024Memory Backdoor Attacks on Neural Networkscited by 3
  17. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  18. 2026Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approachcited by 3
  19. 2024HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantizationcited by 2
  20. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  21. 2025Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updatescited by 2
  22. 2025FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Graftingcited by 2
  23. 2026FedKRSO: Communication and Memory Efficient Federated Fine-Tuning of Large Language Modelscited by 2
  24. 2026Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRAcited by 2
  25. 2024Understanding the Statistical Accuracy-Communication Trade-off in Personalized Federated Learning with Minimax Guaranteescited by 1
  26. 2024LEGO: Language Model Building Blockscited by 1
  27. 2025Federated Multimodal Learning with Dual Adapters and Selective Pruning for Communication and Computational Efficiencycited by 1
  28. 2025Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Samplingcited by 1
  29. 2025Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Modelscited by 1
  30. 2025FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMscited by 1
  31. 2025FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregationcited by 1
  32. 2025FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraintscited by 1
  33. 2025Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projectioncited by 1
  34. 2026Reconstructing Training Data from Adapter-based Federated Large Language Modelscited by 1
  35. 2026Making Models Unmergeable via Scaling-Sensitive Loss Landscapecited by 1
  36. 2026Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneitycited by 1
  37. 2026Stabilized Fine-Tuning with LoRA in Federated Learning: Mitigating the Side Effect of Client Size and Rank via the Scaling Factorcited by 1
  38. 2026Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learningcited by 1

50 most cited of 93 All 93 (JSON)