mlsanityData to 7 Oct 2026

FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models

arXiv 2310.01467 ↗First posted 2 Oct 2023cs.CL cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLM) have revolutionized the NLP landscape, achieving stellar performances across diverse tasks. These models, while benefiting from vast training data, often require fine-tuning on specific data to cater to distinct downstream tasks. However, this data adaptation process has inherent security and privacy concerns, primarily when leveraging user-generated, device-residing data. Federated learning (FL) provides a solution, allowing collaborative model fine-tuning without centralized data collection. However, applying FL to finetune PLMs is hampered by challenges, including restricted model parameter access, high computational requirements, and communication overheads. This paper introduces Federated Black-box Prompt Tuning (FedBPT), a framework designed to address these challenges. FedBPT does not require the clients to access the model parameters. By focusing on training optimal prompts and utilizing gradient-free optimization methods, FedBPT reduces the number of exchanged variables, boosts communication efficiency, and minimizes computational and storage costs. Experiments highlight the framework's ability to drastically cut communication and memory costs while maintaining competitive performance. Ultimately, FedBPT presents a promising solution for efficient, privacy-preserving fine-tuning of PLM in the age of large language models.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

37 of 53 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2019Advances and Open Problems in Federated Learningcited by 3,021
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
25 more
  1. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2018Federated Learning with Non-IID Datacited by 1,099
  5. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  6. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  7. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  8. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  9. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  10. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  11. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  12. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  13. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  14. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  15. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  16. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  17. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  18. 2020Provable Defense against Privacy Leakage in Federated Learning from Representation Perspectivecited by 94
  19. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  20. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  21. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  22. 2021FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learningcited by 43
  23. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  24. 2022Pretrained Models for Multilingual Federated Learningcited by 30
  25. 2022Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devicescited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 9; 2026: 5.

25 papers.

Sort:

By area:

  1. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  2. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 54
  3. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  4. 2024Federated Large Language Models: Current Progress and Future Directionscited by 22
  5. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  6. 2024When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challengescited by 20
  7. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  8. 2023ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learningcited by 9
  9. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  10. 2025FedPOB: Sample-Efficient Federated Prompt Optimization via Banditscited by 3
  11. 2025Federated Adapter on Foundation Models: An Out-Of-Distribution Approachcited by 2
  12. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
13 more
  1. 2026FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewardscited by 2
  2. 2024FedDTPT: Federated Discrete and Transferable Prompt Tuning for Black-Box Large Language Modelscited by 1
  3. 2025FedOne: Query-Efficient Federated Learning for Black-box Discrete Prompt Learningcited by 1
  4. 2024A Historical Trajectory Assisted Optimization Method for Zeroth-Order Federated Learningcited by 0
  5. 2025FedTDP: A Privacy-Preserving and Unified Framework for Trajectory Data Preparation via Federated Learningcited by 0
  6. 2025LLM-QFL: Distilling Large Language Model for Quantum Federated Learningcited by 0
  7. 2025Federated Foundation Language Model Post-Training Should Focus on Open-Source Modelscited by 0
  8. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoEcited by 0
  9. 2025FedDPG: An Adaptive Yet Efficient Prompt-tuning Approach in Federated Learning Settingscited by 0
  10. 2026ProToken: Token-Level Attribution for Federated Large Language Modelscited by 0
  11. 2026Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMscited by 0
  12. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0
  13. 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approachcited by 0