Process Supervision via Verbal Critique Improves Reasoning in Large Language Models
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Inference-time scaling for LLM reasoning has focused on three axes: chain depth, sample breadth, and learned step-scorers (PRMs). We introduce a fourth axis, granularity of external verbal supervision, via Verbal Process Supervision (VPS), a training-free framework that uses structured natural-language critique from a stronger supervisor to guide an iterative generate-critique-refine loop up to a round budget R. Across GPQA Diamond, AIME 2025, and LiveCodeBench V6 (covering both closed and open models), VPS yields three key results. First, on GPQA Diamond, GPT-5.4 (High) | GPT-5.4 (Low) reaches 94.9% at R=4, surpassing the 94.1% state of the art without gradient updates. Second, on AIME 2025, VPS enables strong weak-actor rescue, boosting scores from 11.7-26.7% to 63.3-90.0% (up to +63.3 points). Third, at matched compute, VPS outperforms Reflexion by +8.5 to +12.1 points and Self-Consistency@5 by +5.0 pp (GPQA) and +8.3 pp (LiveCodeBench), isolating critique granularity as the key driver. Performance scales with the supervisor-actor capability gap (Pearson r=0.90) and degrades when errors are not linguistically expressible (e.g., code synthesis), motivating hybrid verbal-executable methods. These results establish critique granularity as a new axis of inference-time scaling.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 17 of its 17 references that are in the map. First posted on arXiv 23 Apr 2026.
Builds on
17 of 17 references.
Closest ancestors · 7
Same direction, newest first
- 2025s1: Simple test-time scaling31 Jan
- 2024Large Language Models have Intrinsic Self-Correction Ability21 Jun
- 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing19 May
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
- 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
- 2022Large Language Models Can Self-Improve20 Oct
Other references · 10
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
- 2023Reasoning with Language Model is Planning with World Modelcited by 861
- 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
- 2025R-PRM: Reasoning-Driven Process Reward Modelingcited by 38
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critique
- 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
- 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoning
- 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study
- 2026ReEfBench: Quantifying the Reasoning Efficiency of LLMs
- 2025ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models
- 2025Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation
- 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models