mlsanityData to 7 Oct 2026

AgentRM: Enhancing Agent Generalization with Reward Modeling

arXiv 2502.18407 ↗First posted 25 Feb 2025cs.CL cs.AI cs.LGcited by 25

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Existing LLM-based agents have achieved strong performance on held-in tasks, but their generalizability to unseen tasks remains poor. Hence, some recent work focus on fine-tuning the policy model with more diverse tasks to improve the generalizability. In this work, we find that finetuning a reward model to guide the policy model is more robust than directly finetuning the policy model. Based on this finding, we propose AgentRM, a generalizable reward model, to guide the policy model for effective test-time search. We comprehensively investigate three approaches to construct the reward model, including explicit reward modeling, implicit reward modeling and LLM-as-a-judge. We then use AgentRM to guide the answer generation with Best-of-N sampling and step-level beam search. On four types of nine agent tasks, AgentRM enhances the base policy model by 8.8 points on average, surpassing the top general agent by 4.0. Moreover, it demonstrates weak-to-strong generalization, yielding greater improvement of 12.6 on LLaMA-3-70B policy model. As for the specializability, AgentRM can also boost a finetuned policy model and outperform the top specialized agent by 11.4 on three held-in tasks. Further analysis verifies its effectiveness in test-time scaling. Codes will be released to facilitate the research in this area.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 37 of its 57 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

37 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planning17 Feb
  2. 2024From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning6 Nov
  3. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models14 Sep
  4. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents4 Mar
  5. 2023Rational Decision-Making Agent with Internalized Utility Judgment24 Aug

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2024A Survey on LLM-as-a-Judgecited by 1,384
20 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  2. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  3. 2022Large Language Models Can Self-Improvecited by 656
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  5. 2023Augmented Language Models: a Surveycited by 398
  6. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  7. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  8. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  9. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  10. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  11. 2024Free Process Rewards without Process Labelscited by 162
  12. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  13. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  14. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  15. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  16. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  17. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  18. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  19. 2024Learning From Failure: Integrating Negative Examples when Fine-tuning Large Language Models as Agentscited by 49
  20. 2025Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Modelcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 14.

25 papers.

Sort:

By area:

  1. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  2. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 33
  3. 2025DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seekingcited by 20
  4. 2025AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenancecited by 15
  5. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  6. 2025SAND: Boosting LLM Agents with Self-Taught Action Deliberationcited by 14
  7. 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agentscited by 11
  8. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  9. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  10. 2025CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questionscited by 4
  11. 2026Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verificationcited by 4
  12. 2025Enhancing LLMs' Reasoning-Intensive Multimedia Search Capabilities through Fine-Tuning and Reinforcement Learningcited by 3
13 more
  1. 20263D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligencecited by 3
  2. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  3. 2026Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agentscited by 2
  4. 2025AutoContext: Instance-Level Context Learning for LLM Agentscited by 1
  5. 2026IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agentscited by 1
  6. 2026Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQLcited by 1
  7. 2026Tree-based Credit Assignment for Multi-Agent Memory Systemcited by 1
  8. 2026Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agentscited by 1
  9. 2026Task-Adaptive Rubrics for GUI Reward Modelingcited by 1
  10. 2026AgentXRay: White-Boxing Agentic Systems via Workflow Reconstructioncited by 0
  11. 2026Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaborationcited by 0
  12. 2026AutoVerifier: Residual-Guided Non-Parametric Optimization for Reference-Based Answer Verificationcited by 0
  13. 2026JET: Judge-Guided Evolution at Test Time for Agent Programscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentV-RL: Scaling Reward Modeling with Agentic Verifier
  2. 2025Process Reward Models for LLM Agents: Practical Framework and Directions
  3. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill
  4. 2026Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization
  5. 2025DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
  6. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
  7. 2026PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent
  8. 2025OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning