mlsanityData to 7 Oct 2026

Entropy-Regularized Process Reward Model

arXiv 2412.11006 ↗First posted 15 Dec 2024cs.LG cs.CLcited by 38

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have shown promise in performing complex multi-step reasoning, yet they continue to struggle with mathematical reasoning, often making systematic errors. A promising solution is reinforcement learning (RL) guided by reward models, particularly those focusing on process rewards, which score each intermediate step rather than solely evaluating the final outcome. This approach is more effective at guiding policy models towards correct reasoning trajectories. In this work, we propose an entropy-regularized process reward model (ER-PRM) that integrates KL-regularized Markov Decision Processes (MDP) to balance policy optimization with the need to prevent the policy from shifting too far from its initial distribution. We derive a novel reward construction method based on the theoretical results. Our theoretical analysis shows that we could derive the optimal reward model from the initial policy sampling. Our empirical experiments on the MATH and GSM8K benchmarks demonstrate that ER-PRM consistently outperforms existing process reward models, achieving 1% improvement on GSM8K and 2-3% improvement on MATH under best-of-N evaluation, and more than 1% improvement under RLHF. These results highlight the efficacy of entropy-regularization in enhancing LLMs' reasoning capabilities.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 35 references that are in the map. First posted on arXiv 15 Dec 2024.

Builds on

32 of 35 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  3. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  4. 2024Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing1 Feb
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  9. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  10. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  11. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  12. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
14 more
  1. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  2. 2024Iterative Reasoning Preference Optimizationcited by 244
  3. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  4. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  5. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  6. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 156
  7. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  8. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  9. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  10. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  11. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 76
  12. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 71
  13. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  14. 2024Augmenting Math Word Problems via Iterative Question Composingcited by 58

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 24; 2026: 13.

38 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  3. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  4. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  5. 2025On Entropy Control in LLM-RL Algorithmscited by 39
  6. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 37
  7. 2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
  8. 2025Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewardscited by 24
  9. 2025ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shapingcited by 16
  10. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  11. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 14
  12. 2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13
26 more
  1. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 12
  2. 2026Multi-Turn On-Policy Distillation with Prefix Replaycited by 12
  3. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  4. 2025VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detectioncited by 11
  5. 2025More Bang for the Buck: Process Reward Modeling with Entropy-Driven Uncertaintycited by 10
  6. 2025Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Taskscited by 10
  7. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewardscited by 9
  8. 2025Fractured Chain-of-Thought Reasoningcited by 7
  9. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  10. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scalingcited by 5
  11. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 5
  12. 2026PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignmentcited by 5
  13. 2026PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimizationcited by 5
  14. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
  15. 2025VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Trainingcited by 3
  16. 2025What Makes Reasoning Invalid: Echo Reflection Mitigation for Large Language Modelscited by 2
  17. 2026PROMISE: Process Reward Models Unlock Test-Time Scaling Laws in Generative Recommendationscited by 2
  18. 2026Towards Robust Process Reward Modeling via Noise-aware Learningcited by 2
  19. 2026SDR: Set-Distance Rewards for Radiology Report Generationcited by 2
  20. 2026Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimizationcited by 1
  21. 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agentscited by 1
  22. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  23. 2026rePIRL: Learn PRM with Inverse RL for LLM Reasoningcited by 0
  24. 2026GR-Ben: A General Reasoning Benchmark for Evaluating Process Reward Modelscited by 0
  25. 2026From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Modelscited by 0
  26. 2026Reinforcement Learning without Ground-Truth Solutions can Improve LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  2. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  3. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement
  4. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  5. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
  6. 2025RLFR: Extending Reinforcement Learning for LLMs with Flow Environment
  7. 2026Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
  8. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs