mlsanityData to 14 Oct 2026

RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains

arXiv 2605.29156 ↗First posted 27 May 2026cs.LG cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Pointwise reward modeling offers critical signals for LLM post-training, yet struggles with absolute scoring in subjective, non-verifiable settings. Rubric-based methods address this by decomposing evaluation into explicit criteria, but existing approaches typically depend on frontier LLMs and suffer from ties caused by hard Boolean aggregation. We present RUBRIC-ARROW, an alternating framework that jointly trains a rubric generator and a rubric-conditioned judge, with its RL stage using only pairwise preference data. Our method couples a probability-based scoring rule that reduces ties with phase-specific preference-based rewards and an alternating GRPO scheme that together train the pointwise evaluator. Extensive experiments show that RUBRIC-ARROW achieves competitive reward-modeling accuracy and yields consistent gains for downstream policy post-training.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 55 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

52 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric15 Feb
  2. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training2 Feb
  3. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling20 Oct
  4. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment9 Oct
  5. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training25 Sep

Other references · 47

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
35 more
  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  4. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  6. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  7. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  8. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  9. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  10. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  11. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  12. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  13. 2024Generative Reward Modelscited by 148
  14. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  15. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  16. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
  17. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  18. 2024How to Evaluate Reward Models for RLHFcited by 88
  19. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  20. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  21. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  22. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
  23. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  24. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
  25. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
  26. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 51
  27. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  28. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
  29. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 20
  30. 2025UltraIF: Advancing Instruction Following from the Wildcited by 18
  31. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  32. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
  33. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 16
  34. 2025Beyond Correctness: Evaluating Subjective Writing Preferences Across Culturescited by 6
  35. 2025DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluationcited by 5
  2. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  3. 2026Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidencecited by 2
  4. 2026RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generationcited by 1
  5. 2026Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judgescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
  2. 2026Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
  3. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
  4. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
  5. 2026Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewards
  6. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
  7. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
  8. 2026Step-wise Rubric Rewards for LLM Reasoning