mlsanityData to 7 Oct 2026

Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gain

arXiv 2603.17815 ↗First posted 18 Mar 2026cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Understanding and evaluating multi-step reasoning in LLMs at the level of individual steps remains a key challenge. Process reward models (PRMs) provide a solution by scoring each step, enabling fine-grained supervision and improved reliability. However, training them requires costly human annotation or computationally intensive rollout-based labeling. To solve this, we introduce MCNIG, a scalable method for automatically labeling the quality of individual reasoning steps in any verifiable domain. Its step score, net information gain (NetIG), improves upon single-reference information gain (IG) by comparing the most-supported correct answer against the most-supported incorrect one, yielding a robust signal even for long and structured outputs like code and SQL, where IG fails. We show that the signal produced by MCNIG correlates with human judgments of step quality, and we apply MCNIG labels to train PRMs that achieve the best average best-of-K accuracy across eight benchmarks spanning mathematics, code generation, text-to-SQL, and scientific QA. Crucially, MCNIG generates no rollouts, cutting labeling complexity to O(N) and making it up to X times cheaper than rollout-based methods at comparable label quality, which makes large-scale process supervision practical.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 41 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

32 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Process Reinforcement through Implicit Rewards3 Feb
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  3. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  4. 2024Free Process Rewards without Process Labels2 Dec
  5. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  6. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision5 Feb
  7. 2023Let's Verify Step by Step31 May

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  10. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
13 more
  1. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  2. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  4. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  5. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  6. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 90
  7. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 84
  8. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Modelscited by 45
  9. 2025Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropycited by 37
  10. 2024Understanding Chain-of-Thought in LLMs through Information Theorycited by 36
  11. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Modelscited by 28
  12. 2022REV: Information-Theoretic Evaluation of Free-Text Rationalescited by 22
  13. 2024Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoningcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  2. 2025Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
  3. 2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
  4. 2026Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning
  5. 2024Generating Chain-of-Thoughts with a Pairwise-Comparison Approach to Searching for the Most Promising Intermediate Thought
  6. 2026Unsupervised Process Reward Models
  7. 2025Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision
  8. 2025An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning