mlsanityData to 7 Oct 2026

Prompt Optimization with Human Feedback

arXiv 2405.17346 ↗First posted 27 May 2024cs.LG cs.AIcited by 35

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable performances in various tasks. However, the performance of LLMs heavily depends on the input prompt, which has given rise to a number of recent works on prompt optimization. However, previous works often require the availability of a numeric score to assess the quality of every prompt. Unfortunately, when a human user interacts with a black-box LLM, attaining such a score is often infeasible and unreliable. Instead, it is usually significantly easier and more reliable to obtain preference feedback from a human user, i.e., showing the user the responses generated from a pair of prompts and asking the user which one is preferred. Therefore, in this paper, we study the problem of prompt optimization with human feedback (POHF), in which we aim to optimize the prompt for a black-box LLM using only human preference feedback. Drawing inspiration from dueling bandits, we design a theoretically principled strategy to select a pair of prompts to query for preference feedback in every iteration, and hence introduce our algorithm named automated POHF (APOHF). We apply our APOHF algorithm to various tasks, including optimizing user instructions, prompt optimization for text-to-image generative models, and response optimization with human feedback (i.e., further refining the response using a variant of our APOHF). The results demonstrate that our APOHF can efficiently find a good prompt using a small number of preference feedback instances. Our code can be found at urlhttps://github.com/xqlin98/APOHF.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 56 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

47 of 56 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Filtered Direct Preference Optimization22 Apr
  2. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model28 Mar
  3. 2024Direct Preference Optimization with an Offset16 Feb
  4. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  5. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2023PaLM 2 Technical Reportcited by 1,154
27 more
  1. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  2. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  3. 2023Large Language Models as Optimizerscited by 830
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  7. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  8. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  9. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  10. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  11. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 226
  12. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  13. 2020Neural Thompson Samplingcited by 127
  14. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  15. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  16. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  17. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  18. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  19. 2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49
  20. 2024PRewrite: Prompt Rewriting with Reinforcement Learningcited by 43
  21. 2022Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Modelscited by 35
  22. 2024Efficient Prompt Optimization Through the Lens of Best Arm Identificationcited by 35
  23. 2024Localized Zeroth-Order Prompt Optimizationcited by 23
  24. 2021Value-at-Risk Optimization with Gaussian Processescited by 19
  25. 2022Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewardscited by 14
  26. 2024Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesiscited by 7
  27. 2024Online Personalizing White-box LLMs Generation with Neural Banditscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 17; 2026: 13.

35 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025Self-Supervised Prompt Optimizationcited by 37
  3. 2025AutoPDL: Automatic Prompt Optimization for LLM Agentscited by 27
  4. 2024Aviary: training language agents on challenging scientific taskscited by 23
  5. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLMcited by 22
  6. 2025Aligning LLMs by Predicting Preferences from User Writing Samplescited by 11
  7. 2025SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimationcited by 11
  8. 2024Neural Dueling Bandits: Preference-Based Optimization with Human Feedbackcited by 7
  9. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 6
  10. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inferencecited by 5
  11. 2026Modular Prompt Optimization: Optimizing Structured Prompts with Section-Local Textual Gradientscited by 5
  12. 2026MASPOB: Bandit-Based Prompt Optimization for Multi-Agent Systems with Graph Neural Networkscited by 5
23 more
  1. 2025Active Human Feedback Collection via Neural Contextual Dueling Banditscited by 4
  2. 2025PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMscited by 4
  3. 2024PREDICT: Preference Reasoning by Evaluating Decomposed preferences Inferred from Candidate Trajectoriescited by 3
  4. 2025Bayesian Optimization from Human Feedback: Near-Optimal Regret Boundscited by 3
  5. 2025FedPOB: Sample-Efficient Federated Prompt Optimization via Banditscited by 3
  6. 2026MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systemscited by 3
  7. 2025Federated Linear Dueling Banditscited by 2
  8. 2025Online Clustering of Dueling Banditscited by 2
  9. 2025Prompt Optimization with Logged Bandit Datacited by 1
  10. 2025Deploying AI for Signal Processing education: Selected challenges and intriguing opportunitiescited by 1
  11. 2025Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineeringcited by 1
  12. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  13. 2026UPA: Unsupervised Prompt Agent via Tree-Based Search and Selectioncited by 1
  14. 2026PrefPO: Pairwise Preference Prompt Optimizationcited by 1
  15. 2026Efficient Multi-objective Prompt Optimization via Pure-exploration Banditscited by 1
  16. 2026AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systemscited by 1
  17. 2024Dynamic Analysis and Adaptive Discriminator for Fake News Detectioncited by 0
  18. 2025Learning to Rewrite Prompts for Bootstrapping LLMs on Downstream Taskscited by 0
  19. 2026Stochastic Multi-Armed Bandits with Limited Control Variatescited by 0
  20. 2026ALSO: Adversarial Online Strategy Optimization for Social Agentscited by 0
  21. 2026Linear and Neural Dueling Bandits with Delayed Feedbackcited by 0
  22. 2026Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language modelscited by 0
  23. 2026Metacognitive Steering: Learning the Structure of Scientific Judgmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PLHF: Prompt Optimization with Few-Shot Human Feedback
  2. 2024PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Sampling
  3. 2025Rethinking Prompt Optimizers: From Prompt Merits to Optimization
  4. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  5. 2024GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers
  6. 2023Preference Ranking Optimization for Human Alignment
  7. 2024A Survey on Human Preference Learning for Large Language Models
  8. 2025Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings