mlsanityData to 7 Oct 2026

FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models

arXiv 2310.10049 ↗First posted 16 Oct 2023cs.LG cs.AIcited by 62

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs), such as ChatGPT, LLaMA, GLM, and PaLM, have exhibited remarkable performances across various tasks in recent years. However, LLMs face two main challenges in real-world applications. One challenge is that training LLMs consumes vast computing resources, preventing LLMs from being adopted by small and medium-sized enterprises with limited computing resources. Another is that training LLM requires a large amount of high-quality data, which are often scattered among enterprises. To address these challenges, we propose FATE-LLM, an industrial-grade federated learning framework for large language models. FATE-LLM (1) facilitates federated learning for large language models (coined FedLLM); (2) promotes efficient training of FedLLM using parameter-efficient fine-tuning methods; (3) protects the intellectual property of LLMs; (4) preserves data privacy during training and inference through privacy-preserving mechanisms. We release the code of FATE-LLM at https://github.com/FederatedAI/FATE-LLM to facilitate the research of FedLLM and enable a broad range of industrial applications.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

25 of 44 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2019Advances and Open Problems in Federated Learningcited by 3,021
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  11. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
13 more
  1. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  2. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  3. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  4. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  5. 2017Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryptioncited by 174
  6. 2019SecureBoost: A Lossless Federated Learning Frameworkcited by 174
  7. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 77
  8. 2020Federated Mutual Learningcited by 70
  9. 2019Long and Diverse Text Generation with Planning-based Hierarchical Variational Modelcited by 62
  10. 2021FedIPR: Ownership Verification for Federated Deep Neural Network Modelscited by 33
  11. 2020When Homomorphic Encryption Marries Secret Sharing: Secure Large-Scale Sparse Logistic Regression and Applications in Risk Controlcited by 32
  12. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  13. 2021Privacy-preserving Federated Adversarial Domain Adaption over Feature Groups for Interpretabilitycited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 33; 2025: 14; 2026: 10.

62 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  3. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  4. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  5. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 60
  6. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  7. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  8. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 35
  9. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  10. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  11. 2024Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecastingcited by 26
  12. 2024FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Modelscited by 19
38 more
  1. 2023FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clientscited by 18
  2. 2024Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Modelscited by 18
  3. 2024FedPFT: Federated Proxy Fine-Tuning of Foundation Modelscited by 16
  4. 2024Vertical Federated Learning for Effectiveness, Security, Applicability: A Surveycited by 14
  5. 2024Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Modelscited by 12
  6. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 12
  7. 2024Training Machine Learning models at the Edge: A Surveycited by 11
  8. 2024When IoT Meet LLMs: Applications and Challengescited by 11
  9. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  10. 2024Personalized Wireless Federated Learning for Large Language Modelscited by 10
  11. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  12. 2024Casper: Prompt Sanitization for Protecting User Privacy in Web-Based Large Language Modelscited by 8
  13. 2024FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injectioncited by 8
  14. 2024Thinking Forward: Memory-Efficient Federated Finetuning of Language Modelscited by 7
  15. 2025Federated In-Context Learning: Iterative Refinement for Improved Answer Qualitycited by 7
  16. 2025A Comprehensive Data-centric Overview of Federated Graph Learningcited by 7
  17. 2025AugFL: Augmenting Federated Learning with Pretrained Modelscited by 6
  18. 2024Federated Co-tuning Framework for Large and Small Language Modelscited by 5
  19. 2024Federated Learning and RAG Integration: A Scalable Approach for Medical Large Language Modelscited by 4
  20. 2025PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillationcited by 4
  21. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  22. 2025Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challengescited by 4
  23. 2023LLMs with User-defined Prompts as Generic Data Operators for Reliable Data Processingcited by 3
  24. 2024FedEval-LLM: Federated Evaluation of Large Language Models on Downstream Tasks with Collective Wisdomcited by 3
  25. 2024CodeCipher: Learning to Obfuscate Source Code Against LLMscited by 3
  26. 2024Memory Backdoor Attacks on Neural Networkscited by 3
  27. 2025SplitFrozen: Split Learning with Device-side Model Frozen for Fine-Tuning LLM on Heterogeneous Resource-Constrained Devicescited by 3
  28. 2024FedBRB: An Effective Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learningcited by 2
  29. 2024FEDSTR: Money-In AI-Out | A Decentralized Marketplace for Federated Learning and LLM Training on the NOSTR Protocolcited by 2
  30. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  31. 2024Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Modelscited by 2
  32. 2024SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learningcited by 1
  33. 2024Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Securitycited by 1
  34. 2024FedCoT: Federated Chain-of-Thought Distillation for Large Language Modelscited by 1
  35. 2024Data Quality Control in Federated Instruction-tuning of Large Language Modelscited by 1
  36. 2024LanFL: Differentially Private Federated Learning with Large Language Models using Synthetic Samplescited by 1
  37. 2025Efficient Split Federated Learning for Large Language Models over Communication Networkscited by 1
  38. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1

50 most cited of 62 All 62 (JSON)