mlsanityData to 14 Oct 2026

OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment

arXiv 2510.07743 ↗First posted 9 Oct 2025cs.CLcited by 132

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Reward modeling lies at the core of reinforcement learning from human feedback (RLHF), yet most existing reward models rely on scalar or pairwise judgments that fail to capture the multifaceted nature of human preferences. Recent studies have explored rubrics-as-rewards (RaR) that uses structured criteria to capture multiple dimensions of response quality. However, producing rubrics that are both reliable and scalable remains a key challenge. In this work, we introduce OpenRubrics, a diverse, large-scale collection of (prompt, rubric) pairs for training rubric-generation and rubric-based reward models. To elicit discriminative and comprehensive evaluation signals, we introduce Contrastive Rubric Generation (CRG), which derives both hard rules (explicit constraints) and principles (implicit qualities) by contrasting preferred and rejected responses. We further remove noisy rubrics via preserving preference-label consistency. Across multiple reward-modeling benchmarks, our rubric-based reward model, Rubric-RM, surpasses strong size-matched baselines by 8.4%. These gains transfer to policy models on instruction-following and biomedical benchmarks.

Key facts. Cited by 132 papers in the map (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 66 of its 93 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

66 of 93 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training25 Sep

Other references · 65

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2025Gemma 3 Technical Reportcited by 1,824
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  11. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
38 more
  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  8. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  9. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  12. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  13. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  14. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  15. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  16. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  17. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  18. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  19. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  20. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  21. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  22. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  23. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  24. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  25. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  26. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  27. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  28. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
  29. 2025Reinforcement Learning with Rubric Anchorscited by 113
  30. 2024Rule Based Rewards for Language Model Safetycited by 108
  31. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  32. 2024Critique-out-Loud Reward Modelscited by 106
  33. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  34. 2024How to Evaluate Reward Models for RLHFcited by 88
  35. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  36. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  37. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  38. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 126.

132 papers.

Sort:

By area:

  1. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  2. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
  3. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  4. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 30
  5. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28
  6. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  7. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
  8. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
  9. 2026ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Rankingcited by 17
  10. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 15
  11. 2026CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Usecited by 14
  12. 2026EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learningcited by 14
38 more
  1. 2026Rubric-based On-policy Distillationcited by 13
  2. 2025AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoningcited by 11
  3. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 11
  4. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRcited by 11
  5. 2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9
  6. 2026ATTNPO: Attention-Guided Process Supervision for Efficient Reasoningcited by 9
  7. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RLcited by 9
  8. 2026Qworld: Question-Specific Evaluation Criteria for LLMscited by 8
  9. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  10. 2026StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Framework with Rubric-based Agentic Reinforcement Learningcited by 7
  11. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 7
  12. 2026SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probingcited by 6
  13. 2026Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproductioncited by 6
  14. 2026P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklistcited by 5
  15. 2026AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judgecited by 5
  16. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  17. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 5
  18. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 5
  19. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluationcited by 4
  20. 2026Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judgescited by 4
  21. 2026Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judgecited by 4
  22. 2026SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utilitycited by 4
  23. 2026Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidancecited by 4
  24. 2026Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewardscited by 4
  25. 2026Reinforcement Learning with Robust Rubric Rewardscited by 4
  26. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  27. 2026ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agentscited by 4
  28. 2026Rubrics as Privileged Information for Open-Ended Generationcited by 4
  29. 2026Best Practice Critic Optimizationcited by 4
  30. 2025ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignmentcited by 3
  31. 2026Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoningcited by 3
  32. 2026The Hitchhikers Guide to Rubric Quality Understanding and Enrichmentcited by 3
  33. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferencescited by 3
  34. 2026Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversationscited by 3
  35. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verificationcited by 3
  36. 2026AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learningcited by 3
  37. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
  38. 2026Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modelingcited by 3

50 most cited of 132 All 132 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
  2. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
  3. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
  4. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling
  5. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategy
  6. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
  7. 2026OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
  8. 2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models