mlsanityData to 14 Oct 2026

Efficient Process Reward Model Training via Active Learning

arXiv 2504.10559 ↗First posted 14 Apr 2025cs.LG cs.AIcited by 16

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Process Reward Models (PRMs) provide step-level supervision to large language models (LLMs), but scaling up training data annotation remains challenging for both humans and LLMs. To address this limitation, we propose an active learning approach, ActPRM, which proactively selects the most uncertain samples for training, substantially reducing labeling costs. During training, we use the PRM to estimate uncertainty after the forward pass, retaining only highly uncertain data. A capable yet costly reasoning model then labels this data. Then we compute the loss with respect to the labels and update the PRM's weights. We compare ActPRM vs. vanilla fine-tuning, on a pool-based active learning setting, demonstrating that ActPRM reduces 50% annotation, but achieving the comparable or even better performance. Beyond annotation efficiency, we further advance the actively trained PRM by filtering over 1M+ math reasoning trajectories with ActPRM, retaining 60% of the data. A subsequent training on this selected dataset yields a new state-of-the-art (SOTA) PRM on ProcessBench (75.0%) and PRMBench (65.5%) compared with same sized models.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 20 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

14 of 20 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification17 Feb
  2. 2025Process Reinforcement through Implicit Rewards3 Feb
  3. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  4. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  5. 2024Free Process Rewards without Process Labels2 Dec
  6. 2024Process Reward Model with Q-Value Rankings15 Oct

Other references · 8

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2022A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglementcited by 87
  5. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  6. 2024Efficient Exploration for LLMscited by 45
  7. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  8. 2024Sample-Efficient Alignment for LLMscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  2. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  3. 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 9
  4. 2025SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learningcited by 6
  5. 2026When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewardscited by 6
  6. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  7. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  8. 2026Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlocited by 5
  9. 2026The Hidden Signal of Verifier Strictness: Controlling and Improving Step-Wise Verification via Selective Latent Steeringcited by 5
  10. 2025OPV: Outcome-based Process Verifier for Efficient Long Chain-of-Thought Verificationcited by 4
  11. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
  12. 2026Training Data Efficiency in Multimodal Process Reward Modelscited by 1
4 more
  1. 2026CIAR: Interval-based Collaborative Decoding for Image Generation Accelerationcited by 0
  2. 2026CURA: Clinical Uncertainty Risk Alignment for Language Model-Based Risk Predictioncited by 0
  3. 2026Process Rewards with Learned Reliabilitycited by 0
  4. 2026Provable Test-Time Scaling for Beam Search in LLM Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unsupervised Process Reward Models
  2. 2025FreePRM: Training Process Reward Models Without Ground Truth Process Labels
  3. 2025Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision
  4. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  5. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  6. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
  7. 2025An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning
  8. 2025More Bang for the Buck: Process Reward Modeling with Entropy-Driven Uncertainty