mlsanityData to 14 Oct 2026

Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL

arXiv 2309.06553 ↗First posted 13 Sep 2023cs.CL cs.AI cs.LGcited by 37

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

In this study, we aim to enhance the arithmetic reasoning ability of Large Language Models (LLMs) through zero-shot prompt optimization. We identify a previously overlooked objective of query dependency in such optimization and elucidate two ensuing challenges that impede the successful and economical design of prompt optimization techniques. One primary issue is the absence of an effective method to evaluate prompts during inference when the golden answer is unavailable. Concurrently, learning via interactions with the LLMs to navigate the expansive natural language prompting space proves to be resource-intensive. To address this, we introduce Prompt-OIRL, which harnesses offline inverse reinforcement learning to draw insights from offline prompting demonstration data. Such data exists as by-products when diverse prompts are benchmarked on open-accessible datasets. With Prompt-OIRL, the query-dependent prompt optimization objective is achieved by first learning an offline reward model. This model can evaluate any query-prompt pairs without accessing LLMs. Subsequently, a best-of-N strategy is deployed to recommend the optimal prompt. Our experimental evaluations across various LLM scales and arithmetic reasoning datasets underscore both the efficacy and economic viability of the proposed approach.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 105 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

60 of 105 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Automatic Chain of Thought Prompting in Large Language Models7 Oct
  2. 2022Large Language Models are Zero-Shot Reasoners24 May
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  9. 2016Generative Adversarial Imitation Learningcited by 1,597
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  13. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  14. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  15. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  16. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  17. 2023Large Language Models as Optimizerscited by 864
  18. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  19. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  20. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  21. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  22. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  23. 2021Implicit Behavioral Cloningcited by 452
  24. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  25. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  26. 2016Solving General Arithmetic Word Problemscited by 360
  27. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  28. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 289
  29. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  30. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  31. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  32. 2023Large Language Model Guided Tree-of-Thoughtcited by 221
  33. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  34. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 214
  35. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  36. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  37. 2020Phasic Policy Gradientcited by 123
  38. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 115

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 15; 2025: 18; 2026: 3.

37 papers.

Sort:

By area:

  1. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  2. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  3. 2024Large Language Models to Enhance Bayesian Optimizationcited by 135
  4. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  5. 2025A Systematic Survey of Automatic Prompt Optimization Techniquescited by 63
  6. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 47
  7. 2024HiGPT: Heterogeneous Graph Language Modelcited by 45
  8. 2024Efficient Prompt Optimization Through the Lens of Best Arm Identificationcited by 37
  9. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  10. 2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
  11. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 13
  12. 2024Towards Hierarchical Multi-Agent Workflows for Zero-Shot Prompt Optimizationcited by 10
25 more
  1. 2024QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learningcited by 10
  2. 2025Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learningcited by 10
  3. 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
  4. 2024Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Modelscited by 7
  5. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  6. 2025No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimizationcited by 7
  7. 2025A Survey of Automatic Prompt Optimization with Instruction-focused Heuristic-based Search Algorithmcited by 6
  8. 2025Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Studycited by 6
  9. 2024Large Language Models for the Automated Analysis of Optimization Algorithmscited by 5
  10. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
  11. 2023When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspectivecited by 4
  12. 2024TestAgent: Automatic Benchmarking and Exploratory Interaction for Evaluating LLMs in Vertical Domainscited by 4
  13. 2025MODS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collectionscited by 4
  14. 2025Fast or Slow? Integrating Fast Intuition and Deliberate Thinking for Enhancing Visual Question Answeringcited by 3
  15. 2024Learning to Generate and Evaluate Fact-checking Explanations with Transformerscited by 2
  16. 2025TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewardscited by 2
  17. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesiscited by 2
  18. 2025Knowing How to Edit: Reliable Evaluation Signals for Diagnosing and Optimizing Prompts at Query Levelcited by 2
  19. 2025Beyond Elicitation: Provision-based Prompt Optimization for Knowledge-Intensive Taskscited by 1
  20. 2025Intelligent Mobile AI-Generated Content Services via Interactive Prompt Engineering and Dynamic Service Provisioningcited by 0
  21. 2025Spatiotemporal Attention-Augmented Inverse Reinforcement Learning for Multi-Agent Task Allocationcited by 0
  22. 2025Informing Acquisition Functions via Foundation Models for Molecular Discoverycited by 0
  23. 2026Automatic Prompt Engineering with No Task Cues and No Tuningcited by 0
  24. 2026Supplement Generation Training for Enhancing Agentic Task Performancecited by 0
  25. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
  2. 2024GRL-Prompt: Towards Knowledge Graph based Prompt Optimization via Reinforcement Learning
  3. 2025PREMISE: Scalable and Strategic Prompt Optimization for Efficient Mathematical Reasoning in Large Models
  4. 2024GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers
  5. 2026Optimizing Prompts for Large Language Models: A Causal Approach
  6. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?
  7. 2026TAPR: Enhancing LLM Performance with a Task-Aware Prompt Rewriter
  8. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study