mlsanityData to 7 Oct 2026

Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

arXiv 2602.14069 ↗First posted 15 Feb 2026cs.CLcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

also in 1 more
Abstract (arXiv)

Scalar reward models compress multi-dimensional human preferences into a single opaque score, creating an information bottleneck that often leads to brittleness and reward hacking in open-ended alignment. We argue that robust alignment for non-verifiable tasks is fundamentally a principle generalization problem: reward should not be a learned function internalized into a judge, but an explicit reasoning process executed under inspectable principles. To operationalize this view, we present the Open Rubric System (OpenRS), a plug-and-play, rubrics-based LLM-as-a-Judge framework built around Pairwise Adaptive Meta-Rubrics (PAMR) and lightweight Pointwise Verifiable Rubrics (PVRs), which provide both hard-constraint guardrails and verifiable reward components when ground-truth or programmatic checks are available. OpenRS uses an explicit meta-rubric -- a constitution-like specification that governs how rubrics are instantiated, weighted, and enforced -- and instantiates adaptive rubrics on the fly by conditioning on the semantic differences between two candidate responses. It then performs criterion-wise pairwise comparisons and aggregates criterion-level preferences externally, avoiding pointwise weighted scalarization while improving discriminability in open-ended settings. To keep principles consistent yet editable across various domains, we introduce a two-level meta-rubric refinement pipeline (automated evolutionary refinement for general principles and a reproducible human-in-the-loop procedure for domain principles), complemented with pointwise verifiable rubrics that act as both guardrails against degenerate behaviors and a source of verifiable reward for objective sub-tasks. Finally, we instantiate OpenRS as reward supervision in pairwise RL training.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 51 references that are in the map. First posted on arXiv 15 Feb 2026.

Builds on

44 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training2 Feb
  2. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling20 Oct
  3. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment9 Oct
  4. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training25 Sep
  5. 2024Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hack9 Oct

Other references · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  9. 2024InternLM2 Technical Reportcited by 391
  10. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  11. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  12. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
27 more
  1. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  2. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  3. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  4. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  5. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  6. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  7. 2024Generative Reward Modelscited by 148
  8. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  9. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  10. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  11. 2025Reinforcement Learning with Rubric Anchorscited by 108
  12. 2024How to Evaluate Reward Models for RLHFcited by 84
  13. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  14. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  15. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  16. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  17. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  18. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 41
  19. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30
  20. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  21. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  22. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 21
  23. 2025Robust Reward Modeling via Causal Rubricscited by 18
  24. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 18
  25. 2026ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Rankingcited by 17
  26. 2025Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoningcited by 10
  27. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 26.

26 papers.

Sort:

By area:

  1. 2026EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubricscited by 12
  2. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
  3. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 4
  4. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  5. 2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Modelscited by 3
  6. 2026Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewardscited by 3
  7. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 3
  8. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
  9. 2026Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Valuescited by 2
  10. 2026Rubric-Guided Process Reward for Stepwise Model Routingcited by 2
  11. 2026ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agentscited by 2
  12. 2026DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Spacecited by 2
14 more
  1. 2026Prompt-Level Reward Specifications for Open-Ended Post-Trainingcited by 1
  2. 2026QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewardscited by 1
  3. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Modelscited by 1
  4. 2026CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judgingcited by 0
  5. 2026Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Shortcited by 0
  6. 2026Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judgescited by 0
  7. 2026Human-Centric Image Captioning with Subject-Centered Spatial Understandingcited by 0
  8. 2026EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learningcited by 0
  9. 2026RLVR2: Reinforcement Learning with Verifiable Rubric-based Rankingcited by 0
  10. 2026RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generationcited by 0
  11. 2026Momentum-Coupled Rubric Adaptation for Detailed Image Captioningcited by 0
  12. 2026Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricscited by 0
  13. 2026MatrixReward: Reward from Rubric Matrix for Open-Ended Generationcited by 0
  14. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Hacking in Rubric-Based Reinforcement Learning
  2. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
  3. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
  4. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
  5. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  6. 2026Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
  7. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategy
  8. 2026MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning