mlsanityData to 14 Oct 2026

Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training

arXiv 2509.21500 ↗First posted 25 Sep 2025cs.LG cs.AIcited by 53

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Reinforcement fine-tuning (RFT) often suffers from reward over-optimization, where a policy model hacks the reward signals to achieve high scores while producing low-quality outputs. Our theoretical analysis shows that the key lies in reward misspecification at the high-reward tail: the inability to reliably distinguish Excellent responses from merely Great ones. This motivate us to focus on the high-reward region. However, such tail examples are scarce under the base LLM. While off-policy exemplars (e.g. from stronger models or rewrites) are easier to obtain, naively training on them yields a misspecified reward for the policy we aim to align. To address this, we study rubric-based rewards. By design, rubrics can leverage off-policy examples while remaining insensitive to their artifacts. To elicit rubrics that capture the high-reward tail, we highlight the importance of distinguishing among great and diverse responses, and introduce a workflow to implement this idea. We empirically demonstrate that rubric-based rewards substantially mitigate reward over-optimization and deliver effective LLM post-training improvements.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 29 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

24 of 29 references.

References · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  10. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  11. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  12. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
12 more
  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  2. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  4. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  5. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  6. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  7. 2025Reinforcement Learning with Rubric Anchorscited by 113
  8. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  9. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  10. 2024InfAlign: Inference-aware language model alignmentcited by 35
  11. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 25
  12. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Modelscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 51.

53 papers.

Sort:

By area:

  1. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  2. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  3. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  4. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
  5. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  6. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  7. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 26
  8. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 26
  9. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 15
  10. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  11. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillationcited by 13
  12. 2026Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Makingcited by 11
38 more
  1. 2026Co-Evolving LLM Evaluators and Policies via DynamicRubriccited by 10
  2. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  3. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 6
  4. 2026Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generationcited by 5
  5. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 5
  6. 2026DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Trainingcited by 5
  7. 2026RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinementcited by 4
  8. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  9. 2026ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agentscited by 4
  10. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferencescited by 3
  11. 2026AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignmentcited by 3
  12. 2026Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modelingcited by 3
  13. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Modelscited by 3
  14. 2026Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigmcited by 2
  15. 2026Deep Research as Rubric for Reinforcement Learningcited by 2
  16. 2026Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalcited by 2
  17. 2026Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidencecited by 2
  18. 2026RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Reviewcited by 2
  19. 2026Momentum-Coupled Rubric Adaptation for Detailed Image Captioningcited by 2
  20. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 1
  21. 2026Prompt-Level Reward Specifications for Open-Ended Post-Trainingcited by 1
  22. 2026Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truthcited by 1
  23. 2026CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Trainingcited by 1
  24. 2026From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mixcited by 1
  25. 2026EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learningcited by 1
  26. 2026RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generationcited by 1
  27. 2026Efficiently Aligning Language Models with Online Natural Language Feedbackcited by 0
  28. 2026CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judgingcited by 0
  29. 2026Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teachercited by 0
  30. 2026CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Datacited by 0
  31. 2026Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classificationcited by 0
  32. 2026From Preferences to Principles: Rubric-Based Alignment for Grounded Knowledge Answerscited by 0
  33. 2026Human-Centric Image Captioning with Subject-Centered Spatial Understandingcited by 0
  34. 2026ShieldVLA: Feasibility-Aware Safety Alignment for Vision-Language-Action Modelscited by 0
  35. 2026DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignmentcited by 0
  36. 2026MatrixReward: Reward from Rubric Matrix for Open-Ended Generationcited by 0
  37. 2026MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learningcited by 0
  38. 2026Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewardscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
  2. 2025ToolRL: Reward is All Tool Learning Needs
  3. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  5. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
  6. 2024Post-hoc Reward Calibration: A Case Study on Length Bias
  7. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  8. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling