mlsanityData to 7 Oct 2026

Towards Building the Federated GPT: Federated Instruction Tuning

arXiv 2305.05644 ↗First posted 9 May 2023cs.CL cs.DC cs.SYcited by 184

Not on the map. Why?

Abstract (arXiv)

While "instruction-tuned" generative large language models (LLMs) have demonstrated an impressive ability to generalize to new tasks, the training phases heavily rely on large amounts of diverse and high-quality instruction data (such as ChatGPT and GPT-4). Unfortunately, acquiring high-quality data, especially when it comes to human-written data, can pose significant challenges both in terms of cost and accessibility. Moreover, concerns related to privacy can further limit access to such data, making the process of obtaining it a complex and nuanced undertaking. Consequently, this hinders the generality of the tuned models and may restrict their effectiveness in certain contexts. To tackle this issue, our study introduces a new approach called Federated Instruction Tuning (FedIT), which leverages federated learning (FL) as the learning framework for the instruction tuning of LLMs. This marks the first exploration of FL-based instruction tuning for LLMs. This is especially important since text data is predominantly generated by end users. Therefore, it is imperative to design and adapt FL approaches to effectively leverage these users' diverse instructions stored on local devices, while preserving privacy and ensuring data security. In the current paper, by conducting widely used GPT-4 auto-evaluation, we demonstrate that by exploiting the heterogeneous and diverse sets of instructions on the client's end with the proposed framework FedIT, we improved the performance of LLMs compared to centralized training with only limited local instructions. Further, in this paper, we developed a Github repository named Shepherd. This repository offers a foundational framework for exploring federated fine-tuning of LLMs using heterogeneous instructions across diverse categories.

Key facts. Cited by 184 papers in the map (data through 7 Oct 2026). Builds on 72 of its 103 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

72 of 103 references.

References · 72

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2019Advances and Open Problems in Federated Learningcited by 3,021
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  3. 2016Deep Learning with Differential Privacycited by 2,682
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  9. 2020Adaptive Federated Optimizationcited by 1,036
  10. 2019Deep Leakage from Gradientscited by 1,004
  11. 2018LEAF: A Benchmark for Federated Settingscited by 780
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  13. 2021GPT Understands, Toocited by 715
  14. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  15. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  16. 2023Instruction Tuning with GPT-4cited by 676
  17. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  18. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  19. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  20. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  21. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  22. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  23. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  24. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  25. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  26. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  27. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  28. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  29. 2020Provable Defense against Privacy Leakage in Federated Learning from Representation Perspectivecited by 94
  30. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
  31. 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
  32. 2019Cognitive Graph for Multi-Hop Reading Comprehension at Scalecited by 65
  33. 2021Towards Fair Federated Learning with Zero-Shot Data Augmentationcited by 64
  34. 2019Active Federated Learningcited by 63
  35. 2020LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasetscited by 60
  36. 2020Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneitycited by 56
  37. 2022LAMP: Extracting Text from Gradients with Language Model Priorscited by 52
  38. 2021FLOP: Federated Learning on Medical Datasets using Partial Networkscited by 42

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 51; 2025: 60; 2026: 59.

184 papers.

Sort:

By area:

  1. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  3. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  4. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 94
  5. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  7. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
  8. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 60
  9. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 59
  10. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
  11. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  12. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 54
38 more
  1. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  2. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  3. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  4. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  5. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 30
  6. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  7. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  8. 2023Integration of Large Language Models and Federated Learningcited by 28
  9. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 26
  10. 2024Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecastingcited by 26
  11. 2023A Survey on Privacy in Graph Neural Networks: Attacks, Preservation, and Applicationscited by 22
  12. 2024Federated Large Language Models: Current Progress and Future Directionscited by 22
  13. 2023Instruction Tuned Models are Quick Learnerscited by 21
  14. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  15. 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 18
  16. 2024Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Modelscited by 18
  17. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
  18. 2023Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Uglycited by 17
  19. 2024Collaborative Learning of Anomalies with Privacy (CLAP) for Unsupervised Video Anomaly Detection: A New Baselinecited by 17
  20. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Modelscited by 16
  21. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  22. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  23. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
  24. 2023NetGPT: A Native-AI Network Architecture Beyond Provisioning Personalized Generative Servicescited by 13
  25. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
  26. 2023Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Modelscited by 12
  27. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  28. 2024DAGER: Exact Gradient Inversion for Large Language Modelscited by 12
  29. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  30. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  31. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  32. 2024Personalized Wireless Federated Learning for Large Language Modelscited by 10
  33. 2024Accelerating Heterogeneous Federated Learning with Closed-form Classifierscited by 10
  34. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
  35. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuningcited by 10
  36. 2025Federated Continual Instruction Tuningcited by 10
  37. 2023Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuningcited by 9
  38. 2024Personalized Collaborative Fine-Tuning for On-Device Large Language Modelscited by 9

50 most cited of 184 All 184 (JSON)