mlsanityData to 14 Oct 2026

Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective

arXiv 2502.10581 ↗First posted 14 Feb 2025cs.LG cs.AI stat.MLcited by 17

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

As large language models have evolved, it has become crucial to distinguish between process supervision and outcome supervision -- two key reinforcement learning approaches to complex reasoning tasks. While process supervision offers intuitive advantages for long-term credit assignment, the precise relationship between these paradigms has remained an open question. Conventional wisdom suggests that outcome supervision is fundamentally more challenging due to the trajectory-level coverage problem, leading to significant investment in collecting fine-grained process supervision data. In this paper, we take steps towards resolving this debate. Our main theorem shows that, under standard data coverage assumptions, reinforcement learning through outcome supervision is no more statistically difficult than through process supervision, up to polynomial factors in horizon. At the core of this result lies the novel Change of Trajectory Measure Lemma -- a technical tool that bridges return-based trajectory measure and step-level distribution shift. Furthermore, for settings with access to a verifier or a rollout capability, we prove that any policy's advantage function can serve as an optimal process reward model, providing a direct connection between outcome and process supervision. These findings suggest that the empirically observed performance gap -- if any -- between outcome and process supervision likely stems from algorithmic limitations rather than inherent statistical difficulties, potentially transforming how we approach data collection and algorithm design for reinforcement learning.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 79 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

35 of 79 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2024Free Process Rewards without Process Labels2 Dec
  3. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  5. 2023Let's Verify Step by Step31 May
  6. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 29

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2024OpenAI o1 System Cardcited by 1,244
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  8. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  9. 2021The Statistical Complexity of Interactive Decision Makingcited by 218
  10. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  11. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  12. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
17 more
  1. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  2. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  3. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  4. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  5. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 79
  6. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  7. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  8. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  9. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  10. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  11. 2023Foundations of Reinforcement Learning and Interactive Decision Makingcited by 33
  12. 2022First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domainscited by 21
  13. 2024Harnessing Density Ratios for Online Reinforcement Learningcited by 15
  14. 2024Scalable Online Exploration via Coverabilitycited by 10
  15. 2024Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedbackcited by 10
  16. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Datacited by 9
  17. 2025Near-optimal Regret Using Policy Optimization in Online MDPs with Aggregate Bandit Feedbackcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 8.

17 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 34
  3. 2025Let Me Think! A Long Chain-of-Thought Can Be Worth Exponentially Many Short Onescited by 13
  4. 2025Distributional AGI Safetycited by 12
  5. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
  6. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  7. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  8. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 5
  9. 2026Post-Training with Policy Gradients: Optimality and the Base Model Barriercited by 3
  10. 2026Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progresscited by 3
  11. 2025AGI Requires a Coordination Layer on Top of Pattern Repositoriescited by 1
  12. 2026When Is Compositional Reasoning Learnable from Verifiable Rewards?cited by 1
5 more
  1. 2025Reasoning without Regretcited by 0
  2. 2026Beyond What Seems Necessary: Hidden Gains from Scaling Training-Time Reasoning Length under Outcome Supervisioncited by 0
  3. 2026The Critical Horizon: Inspection Design Principles for Multi-Stage Operations and Deep Reasoningcited by 0
  4. 2026When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?cited by 0
  5. 2026Reinforcement Learning In Partially Verifiable Environments for Colon Cancer Treatment Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning
  2. 2026Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization
  3. 2026C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learning
  4. 2026Inducing Process Supervision from Outcome-Only Reinforcement Learning
  5. 2026Learning Process Rewards via Reasoning State Propagation
  6. 2025StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models
  7. 2026Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
  8. 2026Discovering Process-Outcome Credit in Multi-Step LLM Reasoning