mlsanityData to 7 Oct 2026

Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

arXiv 2609.38847 ↗First posted 30 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Rubric-based reinforcement learning (Rubric-RL) trains language models where no verifier exists. A judge checks each criterion of a rubric, and the verdicts are aggregated into a reward, most often by a weighted sum. We show that this additive aggregation is the weak point. Under a sum, criteria compensate for one another: a policy that misses the one decision that matters can buy the points back with advice nobody asked for. On clinical consultation, such a policy scores higher and answers worse. Rubric coverage rises while appropriateness on held-out physician criteria falls below the untrained model. The medical criteria are not to blame. Grouped so that they must hold together, the same criteria, unchanged to the word, recover a third of the loss; shorter answers recover almost none. We therefore propose Protocol-level Rubrics (ProRubric), which keeps what the criteria ask for and changes how they are aggregated. It groups a checklist into a few protocol-level dimensions. A dimension counts only when all of its criteria hold and its failure clause does not fire. The grouping is done once, offline, and leaves the optimizer unchanged. ProRubric raises appropriateness by 10.8 points without losing coverage and has the best seven-benchmark average at both scales. Reward validity is set not only by what a rubric verifies, but by how it aggregates. Code is available at https://github.com/Estrellajer/ProRubric

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 48 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

37 of 48 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL12 Aug
  2. 2026RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning10 Aug
  3. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning3 Jun
  4. 2026Reward Hacking in Rubric-Based Reinforcement Learning12 May
  5. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric15 Feb
  6. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks4 Feb
  7. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment9 Oct

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2024Self-Rewarding Language Modelscited by 654
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
18 more
  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  3. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  4. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  5. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  7. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  8. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  9. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  10. 2025Reinforcement Learning with Rubric Anchorscited by 108
  11. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 95
  12. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  13. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 44
  14. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 42
  15. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 22
  16. 2026Step-wise Rubric Rewards for LLM Reasoningcited by 11
  17. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 9
  18. 2026Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
  2. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  3. 2026ConRub-Med: Reinforcement Learning with Consensus Rubrics for Open-Ended Medical Question Answering
  4. 2026OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
  5. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
  6. 2026MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning
  7. 2026QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards
  8. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Models