mlsanityData to 7 Oct 2026

Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modeling

arXiv 2606.07040 ↗First posted 5 Jun 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Open-ended reward modeling requires judges that can follow subtle, domain-specific preferences when verifiable answers are unavailable. Existing rubric-based methods often address this by generating criteria online for each query, but the extra generation step can add inference overhead and produce rigid or misaligned guidance. We introduce Eval-Skill, an exploration-guided method that synthesizes reusable evaluation skills for reward modeling and reframes reward guidance as context evolution rather than parameter training or per-query rubric generation. Using only 100 cases per domain for skill evolution, Eval-Skill synthesizes reusable domain-level evaluation skills through two progressive stages, workflow generation followed by principle generation, with exploration and selection interleaved across both stages. Once generated, a skill is directly injected into the judge context. Across multiple RM benchmarks, Eval-Skill consistently improves diverse judge backbones; on RewardBench 2, it yields significant gains over vanilla judging for each main backbone (+13.44% for Qwen3-8B, and 18.51% for DeepSeek-V4-Flash). Further analyses of evolution-time scaling, generalizability, and transferability show that compact evaluation skills offer an efficient new paradigm for LLM-based evaluation. Code is available at https://github.com/xing-stellus-yue/Eval-Skill.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 46 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

33 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training2 Feb
  2. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment9 Oct

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  5. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  6. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  7. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  8. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  10. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  11. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  12. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 252
19 more
  1. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  2. 2026Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skillscited by 161
  3. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  4. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  5. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 129
  6. 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 129
  7. 2026CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verificationcited by 117
  8. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 80
  9. 2026XSkill: Continual Learning from Experience and Skills in Multimodal Agentscited by 71
  10. 2026SkillX: Automatically Constructing Skill Knowledge Bases for Agentscited by 69
  11. 2025Baichuan-M2: Scaling Medical Capability with Large Verifier Systemcited by 55
  12. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 46
  13. 2026SkillOS: Learning Skill Curation for Self-Evolving Agentscited by 45
  14. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 33
  15. 2026SkillOrchestra: Learning to Route Agents via Skill Transfercited by 22
  16. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  17. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 10
  18. 2026EffiSkill: Agent Skill Based Automated Code Efficiency Optimizationcited by 9
  19. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation
  2. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
  3. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
  4. 2026GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
  5. 2026Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
  6. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models
  7. 2025Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
  8. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge