Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges
On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification also fits: Process reward models for reasoning
Abstract (arXiv)
Rubric-based reward models (RMs) evaluate responses against explicit criteria. However, existing methods often derive criteria from observed response candidates, introducing response-induced bias that may limit rubric generalizability. Their judges also struggle with mechanical constraints when relying on reading alone, and commonly used pairwise judges provide only relative preferences, complicating their direct use for policy optimization. We propose ReVeR, a pointwise rubric RM that derives hard rules and graded principles by reasoning about the instruction alone and evaluates responses independently through tool-augmented judging. Both components are trained through label-blind distillation, in which preference labels only filter a teacher's outputs, yielding ReVeR-Traces, a large-scale dataset of rubric-reasoning and judging trajectories. Experiments demonstrate that ReVeR outperforms pairwise rubric RMs in average preference accuracy, most clearly on instruction following, as well as in test-time scaling and downstream policy optimization. We further find that generator RL improves preference accuracy while reducing rubric fidelity, and that higher preference accuracy does not necessarily translate into better downstream policies. These findings highlight that rubric fidelity, preference accuracy, and downstream utility are distinct properties, shedding light on future RM research.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 70 of its 86 references that are in the map. First posted on arXiv 13 Oct 2026.
Builds on
70 of 86 references.
Closest ancestors · 5
Same direction, newest first
- 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains27 May
- 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling9 Mar
- 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training2 Feb
- 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment9 Oct
- 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training25 Sep
Other references · 65
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
- 2016Concrete Problems in AI Safetycited by 1,642
38 more
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
- 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
- 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
- 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
- 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
- 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
- 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
- 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
- 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
- 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
- 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
- 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
- 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
- 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
- 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
- 2025Generalizing Verifiable Instruction Followingcited by 194
- 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
- 2023Generative Judge for Evaluating Alignmentcited by 169
- 2024Long-form factuality in large language modelscited by 155
- 2025RM-R1: Reward Modeling as Reasoningcited by 154
- 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
- 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
- 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 148
- 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
- 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
- 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
- 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
- 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
- 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
50 most cited of 65 All 65 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
- 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
- 2026Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance
- 2026Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
- 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
- 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
- 2026AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
- 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning