mlsanityData to 7 Oct 2026

Black-Box Prompt Optimization: Aligning Large Language Models without Model Training

arXiv 2311.04155 ↗First posted 7 Nov 2023cs.CLcited by 122

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models (LLMs) have shown impressive success in various applications. However, these models are often not well aligned with human intents, which calls for additional treatments on them; that is, the alignment problem. To make LLMs better follow user instructions, existing alignment methods primarily focus on further training them. However, the extra training of LLMs is usually expensive in terms of GPU computing; even worse, some LLMs are not accessible for user-demanded training, such as GPTs. In this work, we take a different perspective -- Black-Box Prompt Optimization (BPO) -- to perform alignments. The idea is to optimize user prompts to suit LLMs' input understanding, so as to best realize users' intents without updating LLMs' parameters. BPO leverages human preferences to optimize prompts, thus making it superior to LLM (e.g., ChatGPT) as a prompt engineer. Moreover, BPO is model-agnostic, and the empirical results demonstrate that the BPO-aligned ChatGPT yields a 22% increase in the win rate against its original version and 10% for GPT-4. Notably, the BPO-aligned LLMs can outperform the same models aligned by PPO and DPO, and it also brings additional performance gains when combining BPO with PPO or DPO. Code and datasets are released at https://github.com/thu-coai/BPO.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 47 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

31 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
15 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2021GPT Understands, Toocited by 715
  9. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  11. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  12. 2020Artificial Intelligence, Values and Alignmentcited by 443
  13. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  14. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
  15. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 47; 2025: 46; 2026: 26.

122 papers.

Sort:

By area:

  1. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  2. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  3. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  4. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  5. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  6. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 93
  7. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  8. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  9. 2025Safety in Large Reasoning Models: A Surveycited by 58
  10. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  11. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipelinecited by 47
  12. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
38 more
  1. 2024XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolutioncited by 41
  2. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  3. 2024Are Large Language Models Good Prompt Optimizers?cited by 37
  4. 2025VPO: Aligning Text-to-Video Generation Models with Prompt Optimizationcited by 27
  5. 2025AutoPDL: Automatic Prompt Optimization for LLM Agentscited by 27
  6. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  7. 2024Steering Large Language Models between Code Execution and Textual Reasoningcited by 24
  8. 2024iLLM-TSC: Integration reinforcement learning and large language model for traffic signal control policy improvementcited by 23
  9. 2024SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Modelscited by 23
  10. 2024Aviary: training language agents on challenging scientific taskscited by 23
  11. 2023Align on the Fly: Adapting Chatbot Behavior to Established Normscited by 22
  12. 2024FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearchcited by 22
  13. 2024The Fusion of Large Language Models and Formal Methods for Trustworthy AI Agents: A Roadmapcited by 20
  14. 2025How to Train Your Advisor: Steering Black-Box LLMs with Advisor Modelscited by 20
  15. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Modelscited by 19
  16. 2025Failure-Driven Workflow Refinementcited by 19
  17. 2024FaithLM: Towards Faithful Explanations for Large Language Modelscited by 18
  18. 2024HonestLLM: Toward an Honest and Helpful Large Language Modelcited by 17
  19. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  20. 2024Large Language Models for Constrained-Based Causal Discoverycited by 15
  21. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
  22. 2024IPO: Interpretable Prompt Optimization for Vision-Language Modelscited by 13
  23. 2024LangGPT: Rethinking Structured Reusable Prompt Design Framework for LLMs from the Programming Languagecited by 12
  24. 2024Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithmscited by 12
  25. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  26. 2024GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizerscited by 12
  27. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  28. 2025CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstentioncited by 10
  29. 2024AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Modelscited by 9
  30. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9
  31. 2023Plum: Prompt Learning using Metaheuristiccited by 8
  32. 2024Say More with Less: Understanding Prompt Learning Behaviors through Gist Compressioncited by 8
  33. 2024SCULPT: Systematic Tuning of Long Promptscited by 8
  34. 2024A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challengescited by 8
  35. 2025LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimizationcited by 8
  36. 2024PAS: Data-Efficient Plug-and-Play Prompt Augmentation Systemcited by 7
  37. 2024ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimizationcited by 6
  38. 2024Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teamingcited by 6

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  2. 2024Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  4. 2024Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment
  7. 2025Aligning Multimodal LLM with Human Preference: A Survey
  8. 2024DeAL: Decoding-time Alignment for Large Language Models