mlsanityData to 14 Oct 2026

An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning

arXiv 2503.02382 ↗First posted 4 Mar 2025cs.CL cs.AIcited by 12

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Enhancing the mathematical reasoning capabilities of Large Language Models (LLMs) is of great scientific and practical significance. Researchers typically employ process-supervised reward models (PRMs) to guide the reasoning process, effectively improving the models' reasoning abilities. However, existing methods for constructing process supervision training data, such as manual annotation and per-step Monte Carlo estimation, are often costly or suffer from poor quality. To address these challenges, this paper introduces a framework called EpicPRM, which annotates each intermediate reasoning step based on its quantified contribution and uses an adaptive binary search algorithm to enhance both annotation precision and efficiency. Using this approach, we efficiently construct a high-quality process supervision training dataset named Epic50k, consisting of 50k annotated intermediate steps. Compared to other publicly available datasets, the PRM trained on Epic50k demonstrates significantly superior performance. Getting Epic50k at https://github.com/xiaolizh1/EpicPRM.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 47 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

32 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  3. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  5. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  7. 2023Let's Verify Step by Step31 May
  8. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  12. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
12 more
  1. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  2. 2022Large Language Models Can Self-Improvecited by 664
  3. 2021True Few-Shot Learning with Language Modelscited by 344
  4. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  5. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  6. 2023Chain of Code: Reasoning with a Language Model-Augmented Code Emulatorcited by 123
  7. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 109
  8. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
  9. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  10. 2025A Survey of Multilingual Reasoning in Language Modelscited by 33
  11. 2025R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learningcited by 21
  12. 2024Markov Chain of Thought for Efficient Mathematical Reasoningcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  3. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
  4. 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discoverycited by 23
  5. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoningcited by 19
  6. 2026Efficient Process Reward Modeling via Contrastive Mutual Informationcited by 4
  7. 2026LLM Reasoning with Process Rewards for Outcome-Guided Stepscited by 3
  8. 2026Towards Robust Process Reward Modeling via Noise-aware Learningcited by 2
  9. 2026Training Data Efficiency in Multimodal Process Reward Modelscited by 1
  10. 2026Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewardscited by 1
  11. 2026Deep Interaction: An Efficient Human-AI Interaction Method for Large Reasoning Modelscited by 0
  12. 2026When Tokenization is Secretly Output Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning
  2. 2025Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning
  3. 2025Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
  4. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  5. 2025From Static to Dynamic: Adaptive Monte Carlo Search for Mathematical Process Supervision
  6. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  7. 2025Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning
  8. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling