mlsanityData to 14 Oct 2026

Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges

arXiv 2610.18766 ↗First posted 13 Oct 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification also fits: Process reward models for reasoning

Abstract (arXiv)

Rubric-based reward models (RMs) evaluate responses against explicit criteria. However, existing methods often derive criteria from observed response candidates, introducing response-induced bias that may limit rubric generalizability. Their judges also struggle with mechanical constraints when relying on reading alone, and commonly used pairwise judges provide only relative preferences, complicating their direct use for policy optimization. We propose ReVeR, a pointwise rubric RM that derives hard rules and graded principles by reasoning about the instruction alone and evaluates responses independently through tool-augmented judging. Both components are trained through label-blind distillation, in which preference labels only filter a teacher's outputs, yielding ReVeR-Traces, a large-scale dataset of rubric-reasoning and judging trajectories. Experiments demonstrate that ReVeR outperforms pairwise rubric RMs in average preference accuracy, most clearly on instruction following, as well as in test-time scaling and downstream policy optimization. We further find that generator RL improves preference accuracy while reducing rubric fidelity, and that higher preference accuracy does not necessarily translate into better downstream policies. These findings highlight that rubric fidelity, preference accuracy, and downstream utility are distinct properties, shedding light on future RM research.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 70 of its 86 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

70 of 86 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains27 May
  2. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling9 Mar
  3. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training2 Feb
  4. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment9 Oct
  5. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training25 Sep

Other references · 65

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  12. 2016Concrete Problems in AI Safetycited by 1,642
38 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  5. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  9. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  10. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  11. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  12. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  13. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  14. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  15. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  16. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  17. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  18. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  19. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  20. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  21. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  22. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  23. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  24. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  25. 2025Generalizing Verifiable Instruction Followingcited by 194
  26. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  27. 2023Generative Judge for Evaluating Alignmentcited by 169
  28. 2024Long-form factuality in large language modelscited by 155
  29. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  30. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  31. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  32. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 148
  33. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  34. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  35. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  36. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  37. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  38. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78

50 most cited of 65 All 65 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
  2. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  3. 2026Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance
  4. 2026Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
  5. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
  6. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
  7. 2026AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
  8. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning