mlsanityData to 7 Oct 2026

Process Reward Models for LLM Agents: Practical Framework and Directions

arXiv 2502.10325 ↗First posted 14 Feb 2025cs.LG cs.AIcited by 81

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

We introduce Agent Process Reward Models (AgentPRM), a simple and scalable framework for training LLM agents to continually improve through interactions. AgentPRM follows a lightweight actor-critic paradigm, using Monte Carlo rollouts to compute reward targets and optimize policies. It requires minimal modifications to existing RLHF pipelines, making it easy to integrate at scale. Beyond AgentPRM, we propose InversePRM, which learns process rewards directly from demonstrations without explicit outcome supervision. We also explore key challenges and opportunities, including exploration, process reward shaping, and model-predictive reasoning. We evaluate on ALFWorld benchmark, show that small 3B models trained with AgentPRM and InversePRM outperform strong GPT-4o baselines, and analyze test-time scaling, reward hacking, and more. Our code is available at: https://github.com/sanjibanc/agent_prm.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 55 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

28 of 55 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  3. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  4. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  5. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning16 Oct
  6. 2023Let's Verify Step by Step31 May
  7. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  11. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
8 more
  1. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  2. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  3. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  4. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  5. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  6. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  7. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  8. 2024Better than Your Teacher: LLM Agents that learn from Privileged AI Feedbackcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 36; 2026: 45.

81 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  6. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
  7. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  8. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
  9. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  10. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  11. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  12. 2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 41
38 more
  1. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 37
  2. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  3. 2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewardscited by 23
  4. 2025SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graphcited by 22
  5. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 20
  6. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 20
  7. 2025Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planningcited by 15
  8. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  9. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  10. 2025CTRLS: Chain-of-Thought Reasoning via Latent State-Transitioncited by 13
  11. 2025Reinforcement Learning for Machine Learning Engineering Agentscited by 13
  12. 2026Verified Critical Step Optimization for LLM Agentscited by 13
  13. 2025Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agentscited by 11
  14. 2026Wink: Recovering from Misbehaviors in Coding Agentscited by 11
  15. 2026SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditingcited by 11
  16. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agentscited by 10
  17. 2025EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Modelscited by 9
  18. 2025Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learningcited by 9
  19. 2025Can Knowledge-Graph-based Retrieval Augmented Generation Really Retrieve What You Need?cited by 8
  20. 2026SEAGym: An Evaluation Environment for Self-Evolving LLM Agentscited by 8
  21. 2025GRAIL:Learning to Interact with Large Knowledge Graphs for Retrieval Augmented Reasoningcited by 7
  22. 2026HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learningcited by 7
  23. 2026REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Tracescited by 7
  24. 2025Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patternscited by 5
  25. 2025PRInTS: Reward Modeling for Long-Horizon Information Seekingcited by 5
  26. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  27. 2025Beyond Policy Optimization: A Data Curation Flywheel for Sparse-Reward Long-Horizon Planningcited by 4
  28. 2025Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agentscited by 4
  29. 2026TikArt: Stabilizing Aperture-Guided Fine-Grained Visual Reasoning with Reinforcement Learningcited by 4
  30. 2026Learning to Retrieve: Dual-Level Long-Term Memory for Text-to-SQL Agentscited by 4
  31. 2026AdaMEM: Test-Time Adaptive Memory for Language Agentscited by 4
  32. 2026ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agentscited by 4
  33. 2026Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimizationcited by 4
  34. 2025To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learningcited by 3
  35. 2026ReRec: Reasoning-Augmented LLM-based Recommendation Assistant via Reinforcement Fine-tuningcited by 3
  36. 2026TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agentscited by 3
  37. 2026Self-evolving LLM agents with in-distribution Optimizationcited by 3
  38. 2026STAMP: Provenance-Guided Credit Assignment for Deep Search Agentscited by 3

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress
  2. 2026AgentV-RL: Scaling Reward Modeling with Agentic Verifier
  3. 2025RRO: LLM Agent Optimization Through Rising Reward Trajectories
  4. 2025AgentRM: Enhancing Agent Generalization with Reward Modeling
  5. 2026ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents
  6. 2026RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Models
  7. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
  8. 2025MASPRM: Multi-Agent System Process Reward Model