mlsanityData to 7 Oct 2026

Inference-Time Scaling for Generalist Reward Modeling

arXiv 2504.02495 ↗First posted 3 Apr 2025cs.CL cs.AI cs.LGcited by 233

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale. Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that textitproper learning methods could enable effective inference-time scalability. A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules. In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the inference-time scalability of generalist RM, and further, how to improve the effectiveness of performance-compute scaling with proper learning methods. For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and potential for inference-time scaling. For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in DeepSeek-GRM models. Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide voting process for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and could achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models are released at Hugging Face and ModelScope.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 85 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

26 of 85 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 24

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2024OpenAI o1 System Cardcited by 1,220
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  9. 2024DeepSeek-V3 Technical Reportcited by 724
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  11. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  12. 2024InternLM2 Technical Reportcited by 391
12 more
  1. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  2. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
  3. 2024Generative Reward Modelscited by 148
  4. 2024Critique-out-Loud Reward Modelscited by 105
  5. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
  6. 2024Self-Taught Evaluatorscited by 70
  7. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  8. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 35
  9. 2025Atla Selene Mini: A General Purpose Evaluation Modelcited by 27
  10. 2025PairJudge RM: Perform Best-of-N Sampling with Knockout Tournamentcited by 21
  11. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluatorscited by 13
  12. 2024Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 137; 2026: 96.

233 papers.

Sort:

By area:

  1. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  3. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  5. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  6. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  7. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  8. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  9. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 118
  10. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  11. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  12. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 82
38 more
  1. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 75
  2. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  3. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  4. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  5. 2025RewardDance: Reward Scaling in Visual Generationcited by 63
  6. 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modelingcited by 61
  7. 2025Reinforcement Learning from Human Feedbackcited by 53
  8. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 51
  9. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  10. 2025Reward Reasoning Modelcited by 47
  11. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  12. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
  13. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  14. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 43
  15. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  16. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 41
  17. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 39
  18. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 39
  19. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  20. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 36
  21. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  22. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
  23. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  24. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  25. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  26. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignmentcited by 25
  27. 2025Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Rewardcited by 25
  28. 2025EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judgecited by 24
  29. 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 24
  30. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 23
  31. 2025O2-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answeringcited by 22
  32. 2025Think-J: Learning to Think for Generative LLM-as-a-Judgecited by 20
  33. 2026GenAgent: Scaling Text-to-Image Generation via Agentic Multimodal Reasoningcited by 20
  34. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 19
  35. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environmentscited by 19
  36. 2025Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Mathcited by 19
  37. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 19
  38. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 18

50 most cited of 233 All 233 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Power of (Approximate) Reward Models for Inference-Time Scaling
  2. 2025Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
  3. 2025Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling
  4. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  5. 2024Self-Generated Critiques Boost Reward Modeling for Language Models
  6. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  7. 2025Generalist Reward Models: Found Inside Large Language Models
  8. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities