mlsanityData to 14 Oct 2026

HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation

arXiv 2610.19184 ↗First posted 13 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce textscHERO (Harnessing Evidence for Refinement and Optimization), which combines offline and test-time harness optimization. At the first stage, method learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, method achieves the best offline performance, improving pass@1 by (7.6) points over the initial prompt, while test-time refinement adds another (2.0) points and improves execution efficiency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

22 of 34 references.

References · 22

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  5. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  6. 2023Teaching Large Language Models to Self-Debugcited by 988
  7. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  8. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  10. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
  11. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  12. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 309
10 more
  1. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  2. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  3. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 163
  4. 2026SkillOpt: Executive Strategy for Self-Evolving Agent Skillscited by 148
  5. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 142
  6. 2024StreamBench: Towards Benchmarking Continuous Improvement of Language Agentscited by 47
  7. 2024ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?cited by 27
  8. 2024StraGo: Harnessing Strategic Guidance for Prompt Optimizationcited by 17
  9. 2025DSCodeBench: A Realistic Benchmark for Data Science Code Generationcited by 17
  10. 2024CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Eracited by 16

Built on by

Not cited yet.