mlsanityData to 7 Oct 2026

InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models

arXiv 2306.03082 ↗First posted 5 Jun 2023cs.AIcited by 68

Not on the map. Why?

Abstract (arXiv)

Large language models~(LLMs) are instruction followers, but it can be challenging to find the best instruction for different situations, especially for black-box LLMs on which backpropagation is forbidden. Instead of directly optimizing the discrete instruction, we optimize a low-dimensional soft prompt applied to an open-source LLM to generate the instruction for the black-box LLM. On each iteration of the proposed method, which we call InstructZero, a soft prompt is converted into an instruction using the open-source LLM, which is then submitted to the black-box LLM for zero-shot evaluation, and the performance is sent to Bayesian optimization to produce new soft prompts improving the zero-shot performance. We evaluate InstructZero on different combinations of open-source LLMs and APIs including Vicuna and ChatGPT. Our results show that InstructZero outperforms SOTA auto-instruction methods across a variety of downstream tasks. Our code and data are publicly available at https://github.com/Lichang-Chen/InstructZero.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

36 of 48 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
24 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2023Teaching Large Language Models to Self-Debugcited by 963
  6. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 762
  7. 2018A Tutorial on Bayesian Optimizationcited by 734
  8. 2021GPT Understands, Toocited by 715
  9. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  10. 2022Large Language Models Can Self-Improvecited by 656
  11. 2018Junction Tree Variational Autoencoder for Molecular Graph Generationcited by 647
  12. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  13. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  14. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  15. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  16. 2013Bayesian Optimization in a Billion Dimensions via Random Embeddingscited by 178
  17. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  18. 2017Stochastic Zeroth-order Optimization in High Dimensionscited by 62
  19. 2018Molecular Hypergraph Grammar with its Application to Molecular Optimizationcited by 56
  20. 2021Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spacescited by 45
  21. 2023When do you need Chain-of-Thought Prompting for ChatGPT?cited by 34
  22. 2023Prompting Language-Informed Distribution for Compositional Zero-Shot Learningcited by 24
  23. 2023Backdoor Learning on Sequence to Sequence Modelscited by 19
  24. 2023PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizercited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 24; 2025: 20; 2026: 16.

68 papers.

Sort:

By area:

  1. 2023Large Language Models as Optimizerscited by 830
  2. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  3. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  4. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 78
  5. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  6. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  7. 2023AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectorscited by 64
  8. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  9. 2024Are Large Language Models Good Prompt Optimizers?cited by 37
  10. 2024Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplarscited by 37
  11. 2024Efficient Prompt Optimization Through the Lens of Best Arm Identificationcited by 35
  12. 2024Prompt Optimization with Human Feedbackcited by 35
38 more
  1. 2024PromptWizard: Task-Aware Prompt Optimization Frameworkcited by 32
  2. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 30
  3. 2023Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformerscited by 29
  4. 2024Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimizationcited by 28
  5. 2024On Discrete Prompt Optimization for Diffusion Modelscited by 28
  6. 2024Large Language Model-Based Evolutionary Optimizer: Reasoning with elitismcited by 27
  7. 2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
  8. 2024Instance-adaptive Zero-shot Chain-of-Thought Promptingcited by 25
  9. 2024Localized Zeroth-Order Prompt Optimizationcited by 23
  10. 2024FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearchcited by 22
  11. 2023AlpaGasus: Training A Better Alpaca with Fewer Datacited by 19
  12. 2024One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Promptscited by 16
  13. 2025MARS: Multi-Agent Adaptive Reasoning with Socratic Guidance for Automated Prompt Optimizationcited by 15
  14. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  15. 2024A Better LLM Evaluator for Text Generation: The Impact of Prompt Output Sequencing and Optimizationcited by 11
  16. 2024PRSA: Prompt Stealing Attacks against Real-World Prompt Servicescited by 9
  17. 2024Learning to Reason via Program Generation, Emulation, and Searchcited by 9
  18. 2024Hyperband-based Bayesian Optimization for Black-box Prompt Selectioncited by 9
  19. 2024Privacy-preserving Fine-tuning of Large Language Models through Flatnesscited by 6
  20. 2025CAPO: Cost-Aware Prompt Optimizationcited by 6
  21. 2025Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Studycited by 6
  22. 2024Model Fusion through Bayesian Optimization in Language Model Fine-Tuningcited by 5
  23. 2026MASPOB: Bandit-Based Prompt Optimization for Multi-Agent Systems with Graph Neural Networkscited by 5
  24. 2025PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMscited by 4
  25. 2025Every Call is Precious: Global Optimization of Black-Box Functions with Unknown Lipschitz Constantscited by 3
  26. 2025Bayesian Optimization from Human Feedback: Near-Optimal Regret Boundscited by 3
  27. 2024Language Model Prompt Selection via Simulation Optimizationcited by 2
  28. 2024FIDAVL: Fake Image Detection and Attribution using Vision-Language Modelcited by 2
  29. 2025Prompt Tuning Decision Transformers with Structured and Scalable Banditscited by 2
  30. 2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
  31. 2025Sharing State Between Prompts and Programscited by 2
  32. 2026SoftSkill: Behavioral Compression for Contextual Adaptationcited by 2
  33. 2025Evolutionary Prompt Optimization Discovers Emergent Multimodal Reasoning Strategies in Vision-Language Modelscited by 1
  34. 2025FedOne: Query-Efficient Federated Learning for Black-box Discrete Prompt Learningcited by 1
  35. 2025Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregationcited by 1
  36. 2025KASPER: Kolmogorov Arnold Networks for Stock Prediction and Explainable Regimescited by 1
  37. 2025Deploying AI for Signal Processing education: Selected challenges and intriguing opportunitiescited by 1
  38. 2025DRO-InstructZero: Distributionally Robust Prompt Optimization for Large Language Modelscited by 1

50 most cited of 68 All 68 (JSON)