mlsanityData to 7 Oct 2026

Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling

arXiv 2607.01830 ↗First posted 2 Jul 2026cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks. Rubric-based judges offer a transparent way to decompose such judgments into explicit evaluation criteria, but existing annotation-free rubric generators typically rely on a single generic evaluator. As a result, they may overlook important dimensions of human preference, a failure mode we term dimensional blind spots. To address this limitation, we propose Multi-Role Rubric Generation (MRRG), a training-free and reference-free framework that elicits evaluation criteria from multiple complementary roles and consolidates them into an auditable rubric-based scorer. This scorer can be used both to validate pairwise preferences and to provide rewards for GRPO-style Reinforcement Learning with Verifiable Rewards (RLVR). Experiments on preference validation benchmarks show that MRRG consistently outperforms single-role rubric generation baselines across multiple backbone models. Further RLVR experiments demonstrate that MRRG yields a stronger reward signal for improving open-ended generation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 38 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

29 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences15 Apr
  2. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks4 Feb
  3. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment9 Oct
  4. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training25 Sep

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  7. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  10. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  11. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  12. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
13 more
  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  2. 2017Deep Reinforcement Learning: An Overviewcited by 327
  3. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  5. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  6. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  7. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  8. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  9. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  10. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  11. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  12. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  13. 2024How to Evaluate Reward Models for RLHFcited by 84

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidencecited by 0
  2. 2026Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  2. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
  3. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling
  4. 2025Reinforcement Learning with Rubric Anchors
  5. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training
  6. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
  7. 2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models
  8. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Models