mlsanityData to 7 Oct 2026

Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling

arXiv 2510.17314 ↗First posted 20 Oct 2025cs.LG cs.AIcited by 29

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Conventional reward modeling relies on gradient descent over neural weights, creating opaque, data-hungry "black boxes." We propose a paradigm shift from implicit to explicit reward parameterization, recasting optimization from continuous weight spaces to the discrete space of natural language rubrics. We introduce a training-free framework based on iterative rubric learning: it locally induces discriminative criteria via verification-driven refinement, and globally compresses the candidate criteria pool into a compact core set by maximizing an information-theoretic coding rate objective. We organize the compressed core set into a hierarchical rubric structure -- high-level evaluation dimensions supported by concrete verification checks -- serving as an interpretable, portable reward function. Empirically, our approach challenges prevailing data scaling assumptions: using only 70 preference pairs, our rubric-guided judges outperform fully trained reward models on diverse benchmarks. For instance, Qwen3-8B equipped with our learned rubrics achieves 80.91% on RewardBench2, surpassing the specialized Skywork-Reward-V2-Qwen3-8B (78.20%). These results demonstrate that alignment signals are highly compressible and can be effectively captured through explicit symbolic search.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 43 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

38 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment9 Oct

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
25 more
  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  3. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  5. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  7. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  8. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  9. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  10. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  11. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  12. 2023Generative Judge for Evaluating Alignmentcited by 163
  13. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  14. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 141
  15. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  16. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  17. 2024Rule Based Rewards for Language Model Safetycited by 106
  18. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  19. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  20. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  21. 2025Reward Reasoning Modelcited by 47
  22. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 39
  23. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  24. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modellingcited by 21
  25. 2025AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 29.

29 papers.

Sort:

By area:

  1. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  2. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  3. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteriacited by 8
  4. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  5. 2026Qworld: Question-Specific Evaluation Criteria for LLMscited by 6
  6. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 4
  7. 2026SHARP: A Self-Evolving Human-Auditable Rubric Policy for Financial Trading Agentscited by 4
  8. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  9. 2026Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewardscited by 3
  10. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 3
  11. 2026AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Followingcited by 3
  12. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
17 more
  1. 2026Claim-Level Rubric Rewards for Video Caption Reinforcement Learningcited by 3
  2. 2026MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubricscited by 2
  3. 2026Context Pruning for Coding Agents via Multi-Rubric Latent Reasoningcited by 2
  4. 2026PReMISE: Policy Rubrics as Measurement Specifications for LLM Judgescited by 2
  5. 2026ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agentscited by 2
  6. 2026AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignmentcited by 1
  7. 2026Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubricscited by 1
  8. 2026Democratic ICAI: Debating Our Way to Steering Principles from Preferencescited by 1
  9. 2026VGIF-Score: Interpretable and Diagnostic Evaluation of Spatio-Temporal Instruction Following in Video Generationcited by 1
  10. 2026FinProBench: Evaluating Financial AI Agents with Role-Grounded Rubrics Derived from Professional Deliverablescited by 1
  11. 2026BoRP: Bootstrapped Regression Probing for Scalable and Human-Aligned LLM Evaluationcited by 0
  12. 2026Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signalscited by 0
  13. 2026GrowLoop: Self-Evolving Conversation Evaluation Seeded by Humancited by 0
  14. 2026RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Reviewcited by 0
  15. 2026VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learningcited by 0
  16. 2026From Constitutions to Control: Interpretable Rewards for Aligning Language Modelscited by 0
  17. 2026Evaluating Rubric Generation with Interventional Transfercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Rubric Anchors
  2. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
  3. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  4. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
  5. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
  6. 2026Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
  7. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Models
  8. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation