mlsanityData to 14 Oct 2026

StepWiser: Stepwise Generative Judges for Wiser Reasoning

arXiv 2508.19229 ↗First posted 26 Aug 2025cs.AI cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

As models increasingly leverage multi-step reasoning strategies to solve complex problems, supervising the logical validity of these intermediate steps has become a critical research challenge. Process reward models address this by providing step-by-step feedback, but current approaches have two major drawbacks: they typically function as classifiers without providing explanations, and their reliance on supervised fine-tuning with static datasets limits generalization. Inspired by recent advances, we reframe stepwise reward modeling from a classification task to a reasoning task itself. We thus propose a generative judge that reasons about the policy model's reasoning steps (i.e., meta-reasons), outputting thinking tokens before delivering a final verdict. Our model, StepWiser, is trained by reinforcement learning using relative outcomes of rollouts. We show it provides (i) better judgment accuracy on intermediate steps than existing methods; (ii) can be used to improve the policy model at training time; and (iii) improves inference-time search.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 35 of its 38 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

35 of 38 references.

References · 35

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
23 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  2. 2025Process Reinforcement through Implicit Rewardscited by 504
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  5. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  6. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  7. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  8. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  9. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  10. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  13. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  14. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 118
  15. 2025Process Reward Models That Thinkcited by 118
  16. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  17. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 97
  18. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  19. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  20. 2025Diverse Preference Optimizationcited by 52
  21. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 40
  22. 2024Entropy-Regularized Process Reward Modelcited by 38
  23. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 18.

28 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  3. 2025ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaborationcited by 11
  4. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoningcited by 11
  5. 2025ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Modelscited by 10
  6. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  7. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  8. 2026When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Modelscited by 7
  9. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 6
  10. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  11. 2025PRInTS: Reward Modeling for Long-Horizon Information Seekingcited by 5
  12. 2026P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklistcited by 5
16 more
  1. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  2. 2026The Hidden Signal of Verifier Strictness: Controlling and Improving Step-Wise Verification via Selective Latent Steeringcited by 5
  3. 2025Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agentscited by 4
  4. 2026P-GenRM: Personalized Generative Reward Model with Test-time User-based Scalingcited by 4
  5. 2026Efficient Process Reward Modeling via Contrastive Mutual Informationcited by 4
  6. 2026Scaling Medical Reasoning Verification via Tool-Integrated Reinforcement Learningcited by 3
  7. 2026Training Data Efficiency in Multimodal Process Reward Modelscited by 1
  8. 2026rePIRL: Learn PRM with Inverse RL for LLM Reasoningcited by 1
  9. 2026Generative Query Suggestion via Intent Coverage and Query-Level Credit Assignmentcited by 1
  10. 2025Stepwise Think-Critique: Interleaved Reasoning and Self-Critique in a Single LLMcited by 0
  11. 2026Process Rewards with Learned Reliabilitycited by 0
  12. 2026ReCrit: Transition-Aware Reinforcement Learning for Scientific Critic Reasoningcited by 0
  13. 2026Cliff Tokens: Identifying Single-Token Failure Triggers in LLM Mathematical Reasoningcited by 0
  14. 2026Trajectories That Segment Themselves: Agent-Declared Boundaries as a Training Unitcited by 0
  15. 2026Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoningcited by 0
  16. 2026BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning
  2. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
  3. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  4. 2025ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Models
  5. 2026Verified Critical Step Optimization for LLM Agents
  6. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning
  7. 2026Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning
  8. 2026Inducing Process Supervision from Outcome-Only Reinforcement Learning