mlsanityData to 7 Oct 2026

Process reward models for reasoning

576 papers4 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: reasoning, reward, process, training, llms

Key facts. Process reward models for reasoning: 576 papers on arXiv since 2021 (data through 7 Oct 2026). 153 of its 576 mapped papers (27%) cite “Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations” (2023, arXiv 2312.08935). 98 of its 576 mapped papers (17%) cite “The Lessons of Developing Process Reward Models in Mathematical Reasoning” (2025, arXiv 2501.07301). 16 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

16 new papers, 17 Sep – 7 Oct; 4 this week.

  1. This week · 4
  2. BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregation
  3. Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression also fits: Reward modeling and RLHF
  4. Reinforcement Learning for Hierarchical Reasoning Rewards: Minimax-Optimal Rates with Transformers also fits: Reward modeling and RLHF
  5. SIGMA: Self-Improving Alignment Generalization from a Model Spec also fits: Safety alignment and jailbreak attacks
  6. Earlier · 12
  7. Learning Process Rewards via Reasoning State Propagation
  8. LexReward: A Taxonomy-Driven Reward Framework for Legal Language Models also fits: Human values and AI feedback alignment
  9. SemOPT: Fixing Semantic Errors in LLM-based Optimization Modeling via Reward-Guided Search
  10. Inducing Process Supervision from Outcome-Only Reinforcement Learning
  11. FastGuide: Accelerating Reward Guidance for Diffusion Large Language Models also fits: Diffusion and multimodal model alignment
  12. Why Deterministic PRM Guidance Underperforms in Discrete Diffusion Reasoning
All 16
  1. ORPG: Reconciling Multiple Reward Objectives through Objective-wise Policy Gradients also fits: Reward modeling and RLHF
  2. Agentic Multi-Turn Reasoning: A Fairness Approach
  3. From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs also fits: Reward modeling and RLHF
  4. ScopeIF: Improving Scope-Aware Precise Instruction-Following in Large Language Models via Graded Reward Modeling
  5. Rufus-Air: An Open LLM Post-Training Recipe
  6. FLARE: A Full-Lifecycle Dense Supervision Paradigm for Long-Horizon Coding Agents via Generative Reward Model

Foundations

Most cited by papers here

  1. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations2023cited by 153 here
  2. The Lessons of Developing Process Reward Models in Mathematical Reasoning2025cited by 98 here
  3. Solving math word problems with process- and outcome-based feedback2022cited by 86 here
  4. Generative Verifiers: Reward Modeling as Next-Token Prediction2024cited by 55 here
  5. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning2024cited by 55 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2017Deep reinforcement learning from human preferencescited by 59 here
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 43 here
  3. 2020Learning to summarize from human feedbackcited by 36 here
  4. 2022Training language models to follow instructions with human feedbackcited by 188 here
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 48 here
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 46 here
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 49 here
  8. 2024Free Process Rewards without Process Labelscited by 32 here
  9. 2025Process Reinforcement through Implicit Rewardscited by 48 here
15 more it builds on, not in this chain
  1. 2022Solving math word problems with process- and outcome-based feedbackcited by 86 here
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 47 here
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 202 here
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 29 here
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 153 here
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 31 here
  7. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 45 here
  8. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 35 here
  9. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 55 here
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 67 here
  11. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 55 here
  12. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 37 here
  13. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 98 here
  14. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 33 here
  15. 2025Process Reward Models That Thinkcited by 35 here

The 24 most cited of 783.

Browse

30 of 576, ranked 6–35

  1. 6Process Reinforcement through Implicit Rewards202548
  2. 7ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search202445
  3. 8Process Reward Models That Think202535
  4. 9Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs202435
  5. 10GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning202533
  6. 11Free Process Rewards without Process Labels202432
  7. 12PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models202528
  8. 13Process Reward Model with Q-Value Rankings202425
  9. 14VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data202520
  10. 15RM-R1: Reward Modeling as Reasoning202520
All 30
  1. 16Iterative Reasoning Preference Optimization202420
  2. 17Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning202419
  3. 18Let's reward step by step: Step-Level reward model as the Navigators for Reasoning202318
  4. 19VisualPRM: An Effective Process Reward Model for Multimodal Reasoning202517
  5. 20ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs202517
  6. 21Step-level Value Preference Optimization for Mathematical Reasoning202416
  7. 22R-PRM: Reasoning-Driven Process Reward Modeling202515
  8. 23Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains202514
  9. 24Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning202514
  10. 25Entropy-Regularized Process Reward Model202412
  11. 26Process Reward Models for LLM Agents: Practical Framework and Directions202511
  12. 27Reward Reasoning Model202511
  13. 28Web-Shepherd: Advancing PRMs for Reinforcing Web Agents202511
  14. 29Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing202411
  15. 30Advancing LLM Reasoning Generalists with Preference Trees202411
  16. 31On Designing Effective RL Reward at Training Time for LLM Reasoning202411
  17. 32ACECODER: Acing Coder RL via Automated Test-Case Synthesis20259
  18. 33Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards20259
  19. 34Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning20249
  20. 35AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification20258