mlsanityData to 7 Oct 2026

FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model

arXiv 2406.17706 ↗First posted 25 Jun 2024cs.LG cs.CL cs.DCcited by 73

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) show amazing performance on many domain-specific tasks after fine-tuning with some appropriate data. However, many domain-specific data are privately distributed across multiple owners. Thus, this dilemma raises the interest in how to perform LLM fine-tuning in federated learning (FL). However, confronted with limited computation and communication capacities, FL clients struggle to fine-tune an LLM effectively. To this end, we introduce FedBiOT, a resource-efficient LLM fine-tuning approach to FL. Specifically, our method involves the server generating a compressed LLM and aligning its performance with the full model. Subsequently, the clients fine-tune a lightweight yet important part of the compressed model, referred to as an adapter. Notice that as the server has no access to the private data owned by the clients, the data used for alignment by the server has a different distribution from the one used for fine-tuning by clients. We formulate the problem into a bi-level optimization problem to minimize the negative effect of data discrepancy and derive the updating rules for the server and clients. We conduct extensive experiments on LLaMA-2, empirically showing that the adapter has exceptional performance when reintegrated into the global LLM. The results also indicate that the proposed FedBiOT significantly reduces resource consumption compared to existing benchmarks, all while achieving comparable performance levels.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

27 of 48 references.

References · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
15 more
  1. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  2. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  3. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  4. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  5. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  6. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  7. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  8. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  9. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
  10. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 77
  11. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  12. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  13. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergencecited by 42
  14. 2023Joint Prompt Optimization of Stacked LLMs using Variational Inferencecited by 32
  15. 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 42; 2026: 15.

73 papers.

Sort:

By area:

  1. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  2. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  3. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  4. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 30
  5. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directionscited by 25
  6. 2024Federated Large Language Models: Current Progress and Future Directionscited by 22
  7. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  8. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  9. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  10. 2025Can Textual Gradient Work in Federated Learning?cited by 14
  11. 2025PluralLLM: Pluralistic Alignment in LLMs via Federated Learningcited by 14
  12. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
38 more
  1. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  2. 2024When IoT Meet LLMs: Applications and Challengescited by 11
  3. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  4. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
  5. 2024Federated Data-Efficient Instruction Tuning for Large Language Modelscited by 9
  6. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  7. 2024Hierarchical Federated Learning with Multi-Timescale Gradient Correctioncited by 7
  8. 2025Federated In-Context Learning: Iterative Refinement for Improved Answer Qualitycited by 7
  9. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  10. 2025A Comprehensive Data-centric Overview of Federated Graph Learningcited by 7
  11. 2025Talk to Right Specialists: Iterative Routing in Multi-agent Systems for Question Answeringcited by 6
  12. 2026Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalizationcited by 5
  13. 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignmentcited by 5
  14. 2025Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Datacited by 4
  15. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  16. 2024FedSpaLLM: Federated Pruning of Large Language Modelscited by 3
  17. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  18. 2025FedMABench: Benchmarking Mobile Agents on Decentralized Heterogeneous User Datacited by 3
  19. 2025FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregationcited by 3
  20. 2025A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMscited by 3
  21. 2025Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agentscited by 3
  22. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  23. 2025Efficient Federated Fine-Tuning of Large Language Models with Layer Dropoutcited by 2
  24. 2025Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptationcited by 2
  25. 2024Understanding the Statistical Accuracy-Communication Trade-off in Personalized Federated Learning with Minimax Guaranteescited by 1
  26. 2024FedDTPT: Federated Discrete and Transferable Prompt Tuning for Black-Box Large Language Modelscited by 1
  27. 2025A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environmentscited by 1
  28. 2025Blockchain-Enabled Privacy-Preserving Second-Order Federated Edge Learning in Personalized Healthcarecited by 1
  29. 2025FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregationcited by 1
  30. 2025A Survey: Towards Privacy and Security in Mobile Large Language Modelscited by 1
  31. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
  32. 2025CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networkscited by 1
  33. 2025HFL-FlowLLM: Large Language Models for Network Traffic Flow Classification in Heterogeneous Federated Learningcited by 1
  34. 2025PrivTune: Efficient and Privacy-Preserving Fine-Tuning of Large Language Models via Device-Cloud Collaborationcited by 1
  35. 2026APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMscited by 1
  36. 2026FERA: Uncertainty-Aware Federated Reasoning for Large Language Modelscited by 1
  37. 2024FedEP: Tailoring Attention to Heterogeneous Data Distribution with Entropy Pooling for Decentralized Federated Learningcited by 0
  38. 2024FedTLU: Federated Learning with Targeted Layer Updatescited by 0

50 most cited of 73 All 73 (JSON)