mlsanityData to 7 Oct 2026

Black-Box Tuning for Language-Model-as-a-Service

arXiv 2201.03514 ↗First posted 10 Jan 2022cs.CL cs.AIcited by 254

Not on the map. Why?

Abstract (arXiv)

Extremely large pre-trained language models (PTMs) such as GPT-3 are usually released as a service. It allows users to design task-specific prompts to query the PTMs through some black-box APIs. In such a scenario, which we call Language-Model-as-a-Service (LMaaS), the gradients of PTMs are usually unavailable. Can we optimize the task prompts by only accessing the model inference APIs? This paper proposes the black-box tuning framework to optimize the continuous prompt prepended to the input text via derivative-free optimization. Instead of optimizing in the original high-dimensional prompt space, which is intractable for traditional derivative-free optimization, we perform optimization in a randomly generated subspace due to the low intrinsic dimensionality of large PTMs. The experimental results show that the black-box tuning with RoBERTa on a few labeled samples not only significantly outperforms manual prompt and GPT-3's in-context learning, but also surpasses the gradient-based counterparts, i.e., prompt tuning and full model tuning.

Key facts. Cited by 254 papers in the map (data through 7 Oct 2026). Builds on 48 of its 68 references that are in the map. First posted on arXiv 10 Jan 2022.

Builds on

48 of 68 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
36 more
  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  9. 2021GPT Understands, Toocited by 715
  10. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  11. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  13. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  14. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  15. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  16. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  17. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  18. 2021True Few-Shot Learning with Language Modelscited by 343
  19. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  20. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  21. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  22. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  23. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  24. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  25. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  26. 2013Bayesian Optimization in a Billion Dimensions via Random Embeddingscited by 178
  27. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  28. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  29. 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98
  30. 2020Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimizationcited by 89
  31. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  32. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  33. 2021Paradigm Shift in Natural Language Processingcited by 39
  34. 2021Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learningcited by 35
  35. 2017ZOOpt: Toolbox for Derivative-Free Optimizationcited by 12
  36. 2015Improving Object Detection with Deep Convolutional Networks via Bayesian Optimization and Structured Predictioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 24; 2023: 87; 2024: 69; 2025: 52; 2026: 22.

254 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2022A Survey on In-context Learningcited by 717
  3. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  4. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  5. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  6. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  7. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  8. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  9. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  10. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  11. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  12. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
38 more
  1. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  2. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  3. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  4. 2023Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learningcited by 132
  5. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  6. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  7. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  8. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  9. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  10. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  11. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  12. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  13. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
  14. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  15. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  16. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  17. 2024Test-Time Model Adaptation with Only Forward Passescited by 74
  18. 2024Parameter Competition Balancing for Model Mergingcited by 71
  19. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  20. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  21. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  22. 2023CodeIE: Large Code Generation Models are Better Few-Shot Information Extractorscited by 67
  23. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  24. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  25. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 59
  26. 2022BBTv2: Towards a Gradient-Free Future with Large Language Modelscited by 53
  27. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  28. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  29. 2022TEMPERA: Test-Time Prompting via Reinforcement Learningcited by 50
  30. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 50
  31. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  32. 2023Learning from models beyond fine-tuningcited by 47
  33. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  34. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  35. 2023When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitationscited by 41
  36. 2022GPS: Genetic Prompt Search for Efficient Few-shot Learningcited by 37
  37. 2022PromptBoosting: Black-Box Text Classification with Ten Forward Passescited by 36
  38. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36

50 most cited of 254 All 254 (JSON)