mlsanityData to 14 Oct 2026

S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models

arXiv 2509.22099 ↗First posted 26 Sep 2025cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

With the rapid development of large language models (LLMs), generative reward models (GRMs) have been widely adopted for reward modeling and evaluation. Previous studies have primarily focused on training specialized GRMs by optimizing them on preference datasets with the judgment correctness as supervision. While it's widely accepted that GRMs with stronger problem-solving capabilities typically exhibit superior judgment abilities, we first identify a significant solve-to-judge gap when examining individual queries. Specifically, the solve-to-judge gap refers to the phenomenon where GRMs struggle to make correct judgments on some queries (14%-37%), despite being fully capable of solving them. In this paper, we propose the Solve-to-Judge (S2J) approach to address this problem. Specifically, S2J simultaneously leverages both the solving and judging capabilities on a single GRM's output for supervision, explicitly linking the GRM's problem-solving and evaluation abilities during model optimization, thereby narrowing the gap. Our comprehensive experiments demonstrate that S2J effectively reduces the solve-to-judge gap by 16.2%, thereby enhancing the model's judgment performance by 5.8%. Notably, S2J achieves state-of-the-art (SOTA) performance among GRMs built on the same base model while utilizing a significantly smaller training dataset. Moreover, S2J accomplishes this through self-evolution without relying on more powerful external models for distillation.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 61 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

57 of 61 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  2. 2025RewardAnything: Generalizable Principle-Following Reward Models4 Jun
  3. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  4. 2025Think-J: Learning to Think for Generative LLM-as-a-Judge20 May
  5. 2025R3: Robust Rubric-Agnostic Reward Models19 May
  6. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages16 May
  7. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future12 Apr
  8. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  9. 2024Self-Generated Critiques Boost Reward Modeling for Language Models25 Nov
  10. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  11. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  12. 2024How to Evaluate Reward Models for RLHF18 Oct
9 more
  1. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment13 Oct
  2. 2024Generative Reward Models2 Oct
  3. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Data1 Oct
  4. 2024HybridFlow: A Flexible and Efficient RLHF Framework28 Sep
  5. 2024Critique-out-Loud Reward Models21 Aug
  6. 2024Improving Reward Models with Synthetic Critiques31 May
  7. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  8. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2024A Survey on LLM-as-a-Judgecited by 1,423
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
24 more
  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  4. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  5. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  6. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  7. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  8. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  9. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  10. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  11. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  12. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 135
  13. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  14. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 97
  15. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  16. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  17. 2024Evaluating Language Models for Efficient Code Generationcited by 73
  18. 2025Reward Reasoning Modelcited by 47
  19. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 37
  20. 2024Direct Judgement Preference Optimizationcited by 33
  21. 2025DeepCritic: Deliberate Critique with Large Language Modelscited by 17
  22. 2025CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewardscited by 12
  23. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  24. 2025Graders should cheat: privileged information enables expert-level automated evaluationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
  2. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  3. 2026Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidancecited by 4
  4. 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Mathcited by 1
  5. 2026BudgetVerify: Budget-Tiered Verification for Financial QAcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling
  2. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  3. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty
  4. 2026EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
  5. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoning
  6. 2026ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
  7. 2025ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Models
  8. 2025Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness