Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Models
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement-learning training of reasoning LLMs (e.g., GRPO) is expensive and requires a controllable environment, committing every contribution to a full training pipeline. We present EvoResearcher, a training-free, inference-time protocol that adds cost-bounded self-reflection to a single frozen LLM backbone. The protocol iterates generate -> self-critique -> revise until a maximum depth D is reached or the critique returns the CONFIRMED sentinel, an implicit early stop that lets the backbone self-verify its answer under a strict compute budget. Four self-reflective meta-reward components (correctness, efficiency, reflection depth, tool-call diversity) act as design principles instantiated as prompt-level mechanisms, so their benefits accrue with zero gradient updates. We validate the protocol on Big-Bench Hard (100 questions) and establish cross-domain behavior on GSM8K (500) and MATH (500) on the same frozen backbone, with cross-model replication on Qwen2.5-72B. All experiments use pure-reasoning benchmarks; the tool-call diversity component is validated in prompt-level form, and the environment-level and multi-agent extensions are design blueprints left to future work. On clean BBH the protocol does not raise accuracy beyond the 95% Wilson interval; its value is cost-bounded self-verification, with the CONFIRMED early stop terminating 82-88% of items at equal accuracy (about 2.1 generations per question).
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 42 references that are in the map. First posted on arXiv 19 Aug 2026.
Builds on
34 of 42 references.
Closest ancestors · 4
Same direction, newest first
- 2025Discriminative Policy Optimization for Token-Level Reward Models29 May
- 2023Let's Verify Step by Step31 May
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
- 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
Other references · 30
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
18 more
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
- 2025StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Modelscited by 26
- 2024Teaching Models to Balance Resisting and Accepting Persuasioncited by 19
- 2026SWE-TRACE: Optimizing Long-Horizon SWE Agents Through Rubric Process Reward Models and Heuristic Test-Time Scalingcited by 12
- 2025An Adversary-Resistant Multi-Agent LLM System via Credibility Scoringcited by 10
- 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
- 2026LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agentcited by 8
- 2026How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1cited by 6
- 2026Retrospective Progress-Aware Self-Refinement for LLM Agent Trainingcited by 6
- 2026ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learningcited by 5
- 2026How Adversarial Environments Mislead Agentic AI?cited by 4
- 2025A Symbolic Adversarial Learning Framework for Evolving Fake News Generation and Detectioncited by 3
- 2026The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agentscited by 3
- 2026Measuring Epistemic Resilience of LLMs Under Misleading Medical Contextcited by 3
- 2026Closing the Reflection Gap: A Free Calibration Bonus for Agentic RLcited by 3
- 2025DPRM: A Dual Implicit Process Reward Model in Multi-Hop Question Answeringcited by 2
- 2026Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loopscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026A Training-Free Regeneration Paradigm: Contrastive Reflection Memory Guided Self-Verification and Self-Improvement
- 2026Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning
- 2026Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution
- 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
- 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
- 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
- 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
- 2025RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning