mlsanityData to 14 Oct 2026

TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewards

arXiv 2507.18618 ↗First posted 24 Jul 2025cs.CL cs.LGcited by 2

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

Prompt optimization improves the reasoning abilities of large language models (LLMs) without requiring parameter updates to the target model. Following heuristic-based "Think step by step" approaches, the field has evolved in two main directions: while one group of methods uses textual feedback to elicit improved prompts from general-purpose LLMs in a training-free way, a concurrent line of research relies on numerical rewards to train a special prompt model, tailored for providing optimal prompts to the target model. In this paper, we introduce the Textual Reward Prompt framework (TRPrompt), which unifies these approaches by directly incorporating textual feedback into training of the prompt model. Our framework does not require prior dataset collection and is being iteratively improved with the feedback on the generated prompts. When coupled with the capacity of an LLM to internalize the notion of what a "good" prompt is, the high-resolution signal provided by the textual rewards allows us to train a prompt model yielding state-of-the-art query-specific prompts for the problems from the challenging math datasets GSMHard and MATH.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 21 references that are in the map. First posted on arXiv 24 Jul 2025.

Builds on

21 of 21 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL13 Sep
  2. 2022Large Language Models are Zero-Shot Reasoners24 May
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  6. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  7. 2022PAL: Program-aided Language Modelscited by 652
  8. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  9. 2024TextGrad: Automatic "Differentiation" via Textcited by 273
  10. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  11. 2023Large Language Models as Evolutionary Optimizerscited by 124
  12. 2023Learning Performance-Improving Code Editscited by 106
6 more
  1. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 95
  2. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 62
  3. 2022TEMPERA: Test-Time Prompting via Reinforcement Learningcited by 50
  4. 2024Improving Reward Models with Synthetic Critiquescited by 44
  5. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  6. 2024QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learningcited by 10
  2. 2026Prompt Codebooks: Discrete Compositional Optimization for Language Model Instruction Refinementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Better by Comparison: Retrieval-Augmented Contrastive Reasoning for Automatic Prompt Optimization
  2. 2026Optimizing Prompts for Large Language Models: A Causal Approach
  3. 2024GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers
  4. 2023Batch Prompting: Efficient Inference with Large Language Model APIs
  5. 2026Small Language Model Helps Resolve Semantic Ambiguity of LLM Prompt
  6. 2025PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning
  7. 2026TATRA: Training-Free Instance-Adaptive Prompting Through Rephrasing and Aggregation
  8. 2025Self-Supervised Prompt Optimization