Data to 14 Oct 2026

Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems

arXiv 2502.19328 ↗First posted 26 Feb 2025cs.CL cs.AIcited by 51

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more

Abstract (arXiv)

Reward models (RMs) are crucial for the training and inference-time scaling up of large language models (LLMs). However, existing reward models primarily focus on human preferences, neglecting verifiable correctness signals which have shown strong potential in training LLMs. In this paper, we propose agentic reward modeling, a reward system that combines reward models with verifiable correctness signals from different aspects to provide reliable rewards. We empirically implement a reward agent, named RewardAgent, that combines human preference rewards with two verifiable signals: factuality and instruction following, to provide more reliable rewards. We conduct comprehensive experiments on existing reward model benchmarks and inference time best-of-n searches on real-world downstream tasks. RewardAgent significantly outperforms vanilla reward models, demonstrating its effectiveness. We further construct training preference pairs using RewardAgent and train an LLM with the DPO objective, achieving superior performance on various NLP benchmarks compared to conventional reward models. Our codes are publicly released to facilitate further research (https://github.com/THU-KEG/Agentic-Reward-Modeling).

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 55 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

40 of 55 references.

Closest ancestors · 6

Same direction, newest first

Other references · 34

2024The Llama 3 Herd of Modelscited by 14,715
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2020Measuring Massive Multitask Language Understandingcited by 7,963
2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
2024Qwen2.5-Coder Technical Reportcited by 1,203
2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991

22 more

2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
2024DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Datacited by 234
2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
2024Rule Based Rewards for Language Model Safetycited by 108
2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 74
2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
2024Quantile Regression for Distributional Reward Models in RLHFcited by 60
2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 59
2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 28.

51 papers.

Sort:

By area:

2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 97
2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 47
2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 44
2025ChestX-Reasoner: Advancing Radiology Foundation Models with Reasoning through Step-by-Step Verificationcited by 31
2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 28
2025Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Densecited by 20
2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
2026Agent-as-a-Judgecited by 17

38 more

2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
2026AgentV-RL: Scaling Reward Modeling with Agentic Verifiercited by 15
2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoningcited by 13
2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluationcited by 12
2025WildIFEval: Instruction Following in the Wildcited by 11
2025Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modelingcited by 11
2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 10
2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directionscited by 9
2025First Finish Search: Efficient Test-Time Scaling in Large Language Modelscited by 9
2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
2025Code Aesthetics with Agentic Reward Feedbackcited by 6
2026Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluationcited by 6
2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertaintycited by 6
2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 5
2026DR2Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Modelscited by 3
2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
2026Visual-ERM: Reward Modeling for Visual Equivalencecited by 3
2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillationcited by 2
2026Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigmcited by 2
2026StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agentscited by 2
2025Teaching LLM to be Persuasive: Reward-Enhanced Policy Optimization for Alignment from Heterogeneous Rewardscited by 1
2026RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Modelscited by 1
2026StoryAlign: Evaluating and Training Reward Models for Story Generationcited by 1
2026Prompt-Level Reward Specifications for Open-Ended Post-Trainingcited by 1
2026TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AIcited by 1
2025Efficient Mixture-of-Agents Serving via Tree-Structured Routing, Adaptive Pruning, and Dependency-Aware Prefill-Decode Overlapcited by 0
2026SPREG: Structured Plan Repair with Entropy-Guided Test-Time Intervention for Large Language Model Reasoningcited by 0
2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learningcited by 0
2026Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimizationcited by 0
2026Scaling Articulated Rationales for MLLM-based Recommendationcited by 0
2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
2026ScopeIF: Improving Scope-Aware Precise Instruction-Following in Large Language Models via Graded Reward Modelingcited by 0
2026HARPO: Hallucination-Aware Reinforcement Learning for Faithful and Creative Language Generationcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way