mlsanityData to 14 Oct 2026

Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting

arXiv 2610.12601 ↗First posted 8 Oct 2026cs.CLcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents also fits: Web, search, and research agents

also in 1 more
Abstract (arXiv)

Prompt optimization is an attractive alternative to reinforcement learning for adapting language-model agents, since it is cheap and produces human-readable artifacts. On factual tasks, however, it carries a failure mode that scalar-reward reinforcement learning does not. An optimizer that reads the training examples can write their answers into the prompt, so an in-distribution gain may be memorization rather than transferable behaviour. We apply reflective prompt evolution (GEPA) to the two system prompts of a multi-agent deep-research agent and evaluate it under a topic-disjoint split in which no topic is shared across train, validation, and test. Held-out citation reward rises by 0.079, from 0.702 to 0.781, with a bootstrap 95% confidence interval of [0.051, 0.107], and the gain replicates across three seeds (mean 0.107). Nearly half of the winning prompt is a memorized answer key, yet a controlled ablation shows this content earns no reward even on the exact training questions it encodes. The large validation-minus-held-out gap is therefore small-sample selection under a noisy judge, not memorization. Varying the verifier isolates what makes the gain transfer. Decomposed per-dimension feedback is the causal ingredient, and collapsing it to a single scalar reward makes held-out transfer negative. The gain persists under an out-of-family judge and is largely reproduced by a hand-written version of the discovered recipe. Credible evaluation of prompt optimizers therefore requires held-out topics, multiple seeds, and verifier ablations. We release the candidate prompts, lineage diffs, figures, tables, and analysis.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 53 of its 64 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

53 of 64 references.

References · 53

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2016Concrete Problems in AI Safetycited by 1,642
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  10. 2023Large Language Models as Optimizerscited by 864
  11. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  12. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
38 more
  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  2. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
  3. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  4. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  5. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  6. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  7. 2022Defining and Characterizing Reward Hackingcited by 203
  8. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
  9. 2025Generalizing Verifiable Instruction Followingcited by 194
  10. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 163
  11. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 161
  12. 2018Categorizing Variants of Goodhart's Lawcited by 149
  13. 2026SkillOpt: Executive Strategy for Self-Evolving Agent Skillscited by 148
  14. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 44
  15. 2024Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Togethercited by 40
  16. 2024PromptWizard: Task-Aware Prompt Optimization Frameworkcited by 36
  17. 2025Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspectivecited by 34
  18. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  19. 2024Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimizationcited by 29
  20. 2025Deep Research: A Systematic Surveycited by 27
  21. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 26
  22. 2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewardscited by 24
  23. 2026Is Grep All You Need? How Agent Harnesses Reshape Agentic Searchcited by 21
  24. 2025Search Wisely: Mitigating Sub-optimal Agentic Searches By Reducing Uncertaintycited by 17
  25. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 16
  26. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 15
  27. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13
  28. 2026Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agentscited by 12
  29. 2025Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learningcited by 11
  30. 2026Self-Preference Bias in Rubric-Based Evaluation of Large Language Modelscited by 11
  31. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRcited by 11
  32. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RLcited by 9
  33. 2026How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1cited by 8
  34. 2026Chronos: Temporal-Aware Conversational Agents with Structured Event Retrieval for Long-Term Memorycited by 8
  35. 2026Recursive Agent Harnessescited by 6
  36. 2025Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systemscited by 5
  37. 2026MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?cited by 5
  38. 2025FrugalRAG: Less is More in RL Finetuning for Multi-Hop Question Answeringcited by 4

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization
  2. 2026UPA: Unsupervised Prompt Agent via Tree-Based Search and Selection
  3. 2026Coding Agents are Strong Prompt Optimizers
  4. 2025Rethinking the Design of Reinforcement Learning-Based Deep Research Agents
  5. 2026SePO: Self-Evolving Prompt Agent for System Prompt Optimization
  6. 2025Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings
  7. 2025GreenTEA: Gradient Descent with Topic-modeling and Evolutionary Auto-prompting
  8. 2026Debate as Reward: A Multi-Agent Reward System for Scientific Ideation via RL Post-Training