mlsanityData to 7 Oct 2026

OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning

arXiv 2402.06954 ↗First posted 10 Feb 2024cs.LG cs.CL cs.DCcited by 124

Not on the map. Why?

Abstract (arXiv)

Trained on massive publicly available data, large language models (LLMs) have demonstrated tremendous success across various fields. While more data contributes to better performance, a disconcerting reality is that high-quality public data will be exhausted in a few years. In this paper, we offer a potential next step for contemporary LLMs: collaborative and privacy-preserving LLM training on the underutilized distributed private data via federated learning (FL), where multiple data owners collaboratively train a shared model without transmitting raw data. To achieve this, we build a concise, integrated, and research-friendly framework/codebase, named OpenFedLLM. It covers federated instruction tuning for enhancing instruction-following capability, federated value alignment for aligning with human values, and 7 representative FL algorithms. Besides, OpenFedLLM supports training on diverse domains, where we cover 8 training datasets; and provides comprehensive evaluations, where we cover 30+ evaluation metrics. Through extensive experiments, we observe that all FL algorithms outperform local training on training LLMs, demonstrating a clear performance improvement across a variety of settings. Notably, in a financial benchmark, Llama2-7B fine-tuned by applying any FL algorithm can outperform GPT-4 by a significant margin while the model obtained through individual training cannot, demonstrating strong motivation for clients to participate in FL. The code is available at https://github.com/rui-ye/OpenFedLLM.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Builds on 105 of its 145 references that are in the map. First posted on arXiv 10 Feb 2024.

Builds on

105 of 145 references.

References · 105

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,053
38 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  13. 2023Mistral 7Bcited by 3,065
  14. 2019Advances and Open Problems in Federated Learningcited by 3,021
  15. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  16. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  17. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  18. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  19. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  20. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  21. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  22. 2020Extracting Training Data from Large Language Modelscited by 2,144
  23. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  24. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  25. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  26. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  27. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  28. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  29. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  30. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  31. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  32. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  33. 2020Adaptive Federated Optimizationcited by 1,036
  34. 2023BloombergGPT: A Large Language Model for Financecited by 923
  35. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  36. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  37. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  38. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 48; 2026: 40.

124 papers.

Sort:

By area:

  1. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  2. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 60
  3. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  4. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  5. 2024Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecastingcited by 26
  6. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
  7. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  8. 2024Federated Large Language Models: Current Progress and Future Directionscited by 22
  9. 2024Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Modelscited by 18
  10. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  11. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  12. 2025Can Textual Gradient Work in Federated Learning?cited by 14
38 more
  1. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  2. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  3. 2024KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Servercited by 11
  4. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  5. 2024RFLPA: A Robust Federated Learning Framework against Poisoning Attacks with Secure Aggregationcited by 10
  6. 2025MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Userscited by 10
  7. 2025Federated Continual Instruction Tuningcited by 10
  8. 2025Federated Learning for Cross-Domain Data Privacy: A Distributed Approach to Secure Collaborationcited by 9
  9. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  10. 2024Efficient Federated Intrusion Detection in 5G ecosystem using optimized BERT-based modelcited by 8
  11. 2024Large Language Models meet Network Slicing Management and Orchestrationcited by 7
  12. 2024Thinking Forward: Memory-Efficient Federated Finetuning of Language Modelscited by 7
  13. 2024FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Datacited by 7
  14. 2025Federated In-Context Learning: Iterative Refinement for Improved Answer Qualitycited by 7
  15. 2024Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularizationcited by 6
  16. 2025Mitigating Unintended Memorization with LoRA in Federated Learning for LLMscited by 6
  17. 2025One-shot Federated Learning Methods: A Practical Guidecited by 6
  18. 2025LLMs meet Federated Learning for Scalable and Secure IoT Managementcited by 6
  19. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Modelscited by 6
  20. 2025DT-UFC: Universal Large Model Feature Coding via Peaky-to-Balanced Distribution Transformationcited by 6
  21. 2025FedSRD: Sparsify-Reconstruct-Decompose for Communication-Efficient Federated Large Language Models Fine-Tuningcited by 5
  22. 2024CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Modelscited by 4
  23. 2024Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descentcited by 4
  24. 2024Federated Learning and RAG Integration: A Scalable Approach for Medical Large Language Modelscited by 4
  25. 2025Federated Fine-Tuning of Large Language Models: Kahneman-Tversky vs. Direct Preference Optimizationcited by 4
  26. 2025FedMerge: Federated Personalization via Model Mergingcited by 4
  27. 2025Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stagescited by 4
  28. 2024PPFedIT: Towards Privacy-Preserving Federated Instruction Tuning with Few-shot Local Examplescited by 3
  29. 2024Federated and Transfer Learning for Cancer Detection Based on Image Analysiscited by 3
  30. 2024Energy-Efficient Split Learning for Fine-Tuning Large Language Models in Edge Networkscited by 3
  31. 2025Large Language Models for Bioinformaticscited by 3
  32. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  33. 2026Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approachcited by 3
  34. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  35. 2024Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Modelscited by 2
  36. 2024Optimizing Cross-Client Domain Coverage for Federated Instruction Tuning of Large Language Modelscited by 2
  37. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  38. 2025Prada: Black-Box LLM Adaptation with Private Data on Resource-Constrained Devicescited by 2

50 most cited of 124 All 124 (JSON)