# Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers

arXiv: https://arxiv.org/abs/2310.02905 · HTML: https://mlsanity.com/paper/2310.02905/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 57 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Black-Box Tuning for Language-Model-as-a-Service](https://mlsanity.com/paper/2201.03514/) (2022)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers](https://mlsanity.com/paper/2309.08532/) (2023)
- [Reframing Instructional Prompts to GPTk's Language](https://mlsanity.com/paper/2109.07830/) (2021)

## Built on by (most cited first, 30 of 30)

- [PromptWizard: Task-Aware Prompt Optimization Framework](https://mlsanity.com/paper/2405.18369/) (2024)
- [Task Facet Learning: A Structured Approach to Prompt Optimization](https://mlsanity.com/paper/2406.10504/) (2024)
- [PromptSculptor: Multi-Agent Based Text-to-Image Prompt Optimization](https://mlsanity.com/paper/2509.12446/) (2025)
- [CAPO: Cost-Aware Prompt Optimization](https://mlsanity.com/paper/2504.16005/) (2025)
- [A Survey of Automatic Prompt Optimization with Instruction-focused Heuristic-based Search Algorithm](https://mlsanity.com/paper/2502.18746/) (2025)
- [DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning](https://mlsanity.com/paper/2405.14899/) (2024)
- [A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models](https://mlsanity.com/paper/2501.03508/) (2025)
- [PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMs](https://mlsanity.com/paper/2510.25808/) (2025)
- [Every Call is Precious: Global Optimization of Black-Box Functions with Unknown Lipschitz Constants](https://mlsanity.com/paper/2502.04290/) (2025)
- [Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks](https://mlsanity.com/paper/2601.21339/) (2026)
- [Evolving Skill-Structured Attack Memory Enhances LLM Jailbreaking](https://mlsanity.com/paper/2605.29237/) (2026)
- [Cost-Aware Multi-Objective Bandits: Theory and Application to Budgeted LLM Configuration Evaluation](https://mlsanity.com/paper/2608.04333/) (2026)
- [ACING: Actor-Critic for Instruction Learning in Black-Box LLMs](https://mlsanity.com/paper/2411.12736/) (2024)
- [PAFT: Prompt-Agnostic Fine-Tuning](https://mlsanity.com/paper/2502.12859/) (2025)
- [Broaden your SCOPE! Efficient Multi-turn Conversation Planning for LLMs with Semantic Space](https://mlsanity.com/paper/2503.11586/) (2025)
- [Fine-tuning LLMs with variational Bayesian last layer for high-dimensional Bayesian optimization](https://mlsanity.com/paper/2510.01471/) (2025)
- [ECPv2: Fast, Efficient, and Scalable Global Optimization of Lipschitz Functions](https://mlsanity.com/paper/2511.16575/) (2025)
- [Efficient Multi-objective Prompt Optimization via Pure-exploration Bandits](https://mlsanity.com/paper/2605.14553/) (2026)
- [Evolutionary Prompt Optimization Discovers Emergent Multimodal Reasoning Strategies in Vision-Language Models](https://mlsanity.com/paper/2503.23503/) (2025)
- [REMoH: A Reflective Evolution of Multi-objective Heuristics approach via Large Language Models](https://mlsanity.com/paper/2506.07759/) (2025)
- [Deploying AI for Signal Processing education: Selected challenges and intriguing opportunities](https://mlsanity.com/paper/2509.08950/) (2025)
- [A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Bandits](https://mlsanity.com/paper/2601.12945/) (2026)
- [Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judge](https://mlsanity.com/paper/2602.11340/) (2026)
- [BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks](https://mlsanity.com/paper/2605.17000/) (2026)
- [COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimization](https://mlsanity.com/paper/2609.11682/) (2026)
- [Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMs](https://mlsanity.com/paper/2506.21573/) (2025)
- [Uncovering Scaling Laws for Large Language Models via Inverse Problems](https://mlsanity.com/paper/2509.07909/) (2025)
- [ALSO: Adversarial Online Strategy Optimization for Social Agents](https://mlsanity.com/paper/2605.15768/) (2026)
- [Towards Spec Learning: Inference-Time Alignment from Preference Pairs](https://mlsanity.com/paper/2606.24004/) (2026)
- [Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptations](https://mlsanity.com/paper/2610.00928/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown remarkable instruction-following capabilities and achieved impressive performances in various applications. However, the performances of LLMs depend heavily on the instructions given to them, which are typically manually tuned with substantial human efforts. Recent work has used the query-efficient Bayesian optimization (BO) algorithm to automatically optimize the instructions given to black-box LLMs. However, BO usually falls short when optimizing highly sophisticated (e.g., high-dimensional) objective functions, such as the functions mapping an instruction to the performance of an LLM. This is mainly due to the limited expressive power of the Gaussian process (GP) which is used by BO as a surrogate to model the objective function. Meanwhile, it has been repeatedly shown that neural networks (NNs), especially pre-trained transformers, possess strong expressive power and can model highly complex functions. So, we adopt a neural bandit algorithm which replaces the GP in BO by an NN surrogate to optimize instructions for black-box LLMs. More importantly, the neural bandit algorithm allows us to naturally couple the NN surrogate with the hidden representation learned by a pre-trained transformer (i.e., an open-source LLM), which significantly boosts its performance. These motivate us to propose our INSTruction optimization usIng Neural bandits Coupled with Transformers (INSTINCT) algorithm. We perform instruction optimization for ChatGPT and use extensive experiments to show that INSTINCT consistently outperforms baselines in different tasks, e.g., various instruction induction tasks and the task of improving zero-shot chain-of-thought instructions. Our code is available at https://github.com/xqlin98/INSTINCT.