mlsanityData to 7 Oct 2026

BBTv2: Towards a Gradient-Free Future with Large Language Models

arXiv 2205.11200 ↗First posted 23 May 2022cs.CL cs.AIcited by 53

Not on the map. Why?

Abstract (arXiv)

Most downstream adaptation methods tune all or part of the parameters of pre-trained models (PTMs) through gradient descent, where the tuning cost increases linearly with the growth of the model size. By contrast, gradient-free methods only require the forward computation of the PTM to tune the prompt, retaining the benefits of efficient tuning and deployment. Though, past work on gradient-free tuning often introduces gradient descent to seek a good initialization of prompt and lacks versatility across tasks and PTMs. In this paper, we present BBTv2, an improved version of Black-Box Tuning, to drive PTMs for few-shot learning. We prepend continuous prompts to every layer of the PTM and propose a divide-and-conquer gradient-free algorithm to optimize the prompts at different layers alternately. Extensive experiments across various tasks and PTMs show that BBTv2 can achieve comparable performance to full model tuning and state-of-the-art parameter-efficient methods (e.g., Adapter, LoRA, BitFit, etc.) under few-shot settings while maintaining much fewer tunable parameters.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

40 of 56 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
28 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2019How Can We Know What Language Models Know?cited by 953
  11. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  13. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  14. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  15. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  16. 2021True Few-Shot Learning with Language Modelscited by 343
  17. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  18. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  19. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  20. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  21. 2013Bayesian Optimization in a Billion Dimensions via Random Embeddingscited by 178
  22. 2015High Dimensional Bayesian Optimisation and Bandits via Additive Modelscited by 137
  23. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  24. 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98
  25. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  26. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  27. 2021Paradigm Shift in Natural Language Processingcited by 39
  28. 2021Derivative-Free Reinforcement Learning: A Reviewcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 19; 2024: 13; 2025: 11; 2026: 7.

53 papers.

Sort:

By area:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  3. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  4. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  5. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 50
  6. 2023Learning from models beyond fine-tuningcited by 47
  7. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  8. 2022PromptBoosting: Black-Box Text Classification with Ten Forward Passescited by 36
  9. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  10. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 30
  11. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 28
  12. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
38 more
  1. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  2. 2024When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challengescited by 20
  3. 2023Learning to Learn from APIs: Black-Box Data-Free Meta-Learningcited by 18
  4. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14
  5. 2023SPELL: Semantic Prompt Evolution based on a LLMcited by 14
  6. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  7. 2022Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learningcited by 11
  8. 2024Derivative-Free Optimization for Low-Rank Adaptation in Large Language Modelscited by 11
  9. 2025FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speedcited by 11
  10. 2023OpenBox: A Python Toolkit for Generalized Black-box Optimizationcited by 9
  11. 2024Hyperband-based Bayesian Optimization for Black-box Prompt Selectioncited by 9
  12. 2025Test-Time Model Adaptation for Quantized Neural Networkscited by 9
  13. 2023Plum: Prompt Learning using Metaheuristiccited by 8
  14. 2022Decoder Tuning: Efficient Language Understanding as Decodingcited by 6
  15. 2023Continual Learning From a Stream of APIscited by 6
  16. 2025ESSA: Evolutionary Strategies for Scalable Alignmentcited by 6
  17. 2023Black-box Prompt Tuning with Subspace Learningcited by 5
  18. 2023Make Prompt-based Black-Box Tuning Colorful: Boosting Model Generalization from Three Orthogonal Perspectivescited by 5
  19. 2024An Evaluation of Large Language Models in Bioinformatics Researchcited by 5
  20. 2024Black-Box Forgettingcited by 5
  21. 2025EA4LLM: A Gradient-Free Approach to Large Language Model Optimization via Evolutionary Algorithmscited by 5
  22. 2023A Bayesian approach for prompt optimization in pre-trained language modelscited by 4
  23. 2024Robust Adaptation of Foundation Models with Black-Box Visual Promptingcited by 3
  24. 2024Multi-Prompting Decoder Helps Better Language Understandingcited by 2
  25. 2024Sharpness-Aware Black-Box Optimizationcited by 2
  26. 2023Enhancing Black-Box Few-Shot Text Classification with Prompt-Based Data Augmentationcited by 1
  27. 2024Blackbox Adaptation for Medical Image Segmentationcited by 1
  28. 2024CPT: Consistent Proxy Tuning for Black-box Optimizationcited by 1
  29. 2025Learning Evolution via Optimization Knowledge Adaptationcited by 1
  30. 2025SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPCcited by 1
  31. 2026Guided Trajectory Optimization with Sparse Scaling for Test-Time Diffusioncited by 1
  32. 2023Derivative Free Weight-space Ensemblingcited by 0
  33. 2025Stochastic Subspace Descent Accelerated via Bi-fidelity Line Searchcited by 0
  34. 2025MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Promptscited by 0
  35. 2025Transformer Injectivity & Geometric Robustness - Analytic Margins and Bi-Lipschitz Uniformity of Sequence-Level Hidden Statescited by 0
  36. 2026Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuningcited by 0
  37. 2026Position: Vector Prompt Interfaces Should Be Exposed to Enable Customization of Large Language Modelscited by 0
  38. 2026Projection-Free Evolution Strategies for Continuous Prompt Searchcited by 0

50 most cited of 53 All 53 (JSON)