mlsanityData to 7 Oct 2026

Federated Foundation Language Model Post-Training Should Focus on Open-Source Models

arXiv 2505.23593 ↗First posted 29 May 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Post-training of foundation language models has emerged as a promising research domain in federated learning (FL) with the goal to enable privacy-preserving model improvements and adaptations to user's downstream tasks. Recent advances in this area adopt centralized post-training approaches that build upon black-box foundation language models where there is no access to model weights and architecture details. Although the use of black-box models has been successful in centralized post-training, their blind replication in FL raises several concerns. Our opinion is that using black-box models in FL contradicts the core principles of federation such as data privacy and autonomy. In this paper, we critically analyze the usage of black-box models in federated post-training, and provide a detailed account of various aspects of openness and their implications for FL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 63 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

37 of 63 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2021GPT Understands, Toocited by 715
  11. 2021Towards Personalized Federated Learningcited by 462
  12. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
25 more
  1. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  2. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  3. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  4. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
  5. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  6. 2025A Survey on Large Language Models with some Insights on their Capabilities and Limitationscited by 41
  7. 2022Privately Fine-Tuning Large Language Models with Differential Privacycited by 39
  8. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 34
  9. 2023Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspectivecited by 34
  10. 2024A Survey on Efficient Federated Learning Methods for Foundation Model Trainingcited by 29
  11. 2023FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Modelscited by 25
  12. 2024PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuningcited by 22
  13. 2023INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptationcited by 20
  14. 2024The Open Source Advantage in Large Language Models (LLMs)cited by 18
  15. 2022Secure Aggregation for Federated Learning in Flowercited by 15
  16. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
  17. 2025Can Textual Gradient Work in Federated Learning?cited by 14
  18. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  19. 2024Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternativescited by 12
  20. 2023Tunable Soft Prompts are Messengers in Federated Learningcited by 10
  21. 2023ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learningcited by 9
  22. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  23. 2024SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learningcited by 5
  24. 2023FLEdge: Benchmarking Federated Machine Learning Applications in Edge Computing Systemscited by 3
  25. 2024FedDTPT: Federated Discrete and Transferable Prompt Tuning for Black-Box Large Language Modelscited by 1

Built on by

Not cited yet.