mlsanityData to 14 Oct 2026

Tunable Soft Prompts are Messengers in Federated Learning

arXiv 2311.06805 ↗First posted 12 Nov 2023cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources, alleviating privacy concerns that arise from directly sharing local data. However, the lack of model privacy protection in FL becomes an unneglectable challenge, especially when people want to federally finetune models based on a proprietary large language model. In this study, we propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts. These soft prompts, updated and transmitted between the server and clients, assume the role of the global model parameters and serve as messengers to deliver useful knowledge from the local data and global model. As the global model itself is not required to be shared and the local training is conducted based on an auxiliary model with fewer parameters than the global model, the proposed approach provides protection for the global model while reducing communication and computation costs in FL. Extensive experiments show the effectiveness of the proposed approach compared to several baselines. We have released the source code at urlhttps://github.com/alibaba/FederatedScope/tree/fedsp/federatedscope/nlp/fedsp.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 40 of its 50 references that are in the map. First posted on arXiv 12 Nov 2023.

Builds on

40 of 50 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
28 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  9. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  10. 2021Model-Contrastive Federated Learningcited by 766
  11. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  12. 2021GPT Understands, Toocited by 716
  13. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  14. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  15. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  16. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 91
  17. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 78
  18. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  19. 2019Federated Learning with Bayesian Differential Privacycited by 60
  20. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  21. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  22. 2021FedMatch: Federated Learning Over Heterogeneous Question Answering Datacited by 23
  23. 2022Training Mixed-Domain Translation Models via Federated Learningcited by 12
  24. 2022Federated Split BERT for Heterogeneous Text Classificationcited by 11
  25. 2023Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adaptercited by 8
  26. 2022Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learningcited by 4
  27. 2022FedQAS: Privacy-aware machine reading comprehension with federated learningcited by 3
  28. 2022FewFedWeight: Few-shot Federated Learning Framework across Multiple NLP Taskscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 2.

10 papers.

  1. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  2. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  3. 2024Promoting Data and Model Privacy in Federated Learning through Quantized LoRAcited by 8
  4. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  5. 2025Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptationcited by 2
  6. 2026Floe: Federated Specialization for Real-Time LLM-SLM Inferencecited by 2
  7. 2025Enhancing Model Privacy in Federated Learning with Random Masking and Quantizationcited by 1
  8. 2026FERA: Uncertainty-Aware Federated Reasoning for Large Language Modelscited by 1
  9. 2025Mitigating Group-Level Fairness Disparities in Federated Visual Language Modelscited by 0
  10. 2025Federated Foundation Language Model Post-Training Should Focus on Open-Source Modelscited by 0