mlsanityData to 7 Oct 2026

Exploring Reasoning Reward Model for Agents

arXiv 2601.22154 ↗First posted 29 Jan 2026cs.AI cs.CLcited by 13

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 41 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

30 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress11 Nov
  2. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Use7 Oct
  3. 2025Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Tasks29 Sep
  4. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May

Other references · 26

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  8. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  11. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  12. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
14 more
  1. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  2. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 118
  3. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 99
  4. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  5. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  6. 2025RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agentscited by 56
  7. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  8. 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
  9. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  10. 2025Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewardscited by 19
  11. 2025ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shapingcited by 16
  12. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoningcited by 14
  13. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  14. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modelingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Gen-Searcher: Reinforcing Agentic Search for Image Generationcited by 25
  2. 2026OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agentscited by 18
  3. 2026PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalizationcited by 8
  4. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  5. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  6. 2026From Web to Pixels: Bringing Agentic Search into Visual Perceptioncited by 3
  7. 2026SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulationcited by 3
  8. 2026FailureMem: A Failure-Aware Multimodal Framework for Autonomous Software Repaircited by 2
  9. 2026Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQLcited by 1
  10. 2026UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimizationcited by 1
  11. 2026ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capabilitycited by 1
  12. 2026AgenticRL: Agentic Reinforcement Learning with Self-Refinement for Complex UAV Navigationcited by 0
1 more
  1. 2026Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentV-RL: Scaling Reward Modeling with Agentic Verifier
  2. 2025Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning
  3. 2025Demystifying Reinforcement Learning in Agentic Reasoning
  4. 2026RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Models
  5. 2026Verifiable Process Rewards for Agentic Reasoning
  6. 2026Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning
  7. 2025Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents
  8. 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards