mlsanityData to 14 Oct 2026

From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics

arXiv 2610.13375 ↗First posted 9 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective. In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation. Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion-level judgments, and sophisticated score aggregation provides limited additional gains. Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement. Experiments across benchmarks show that ARTS-Hybrid improves both end-to-end evaluation performance and intrinsic rubric quality. Further experiments suggest that, beyond evaluation, rubrics generated by ARTS-Hybrid provide effective signals for model optimization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 30 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

27 of 30 references.

References · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  7. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  8. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  10. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  11. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  12. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
15 more
  1. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  2. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 114
  3. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  4. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  5. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 53
  6. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
  7. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28
  8. 2025Deep Research: A Systematic Surveycited by 27
  9. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 26
  10. 2025InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Trainingcited by 12
  11. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modelingcited by 12
  12. 2026Qworld: Question-Specific Evaluation Criteria for LLMscited by 8
  13. 2026Optimsyn: Influence-Guided Rubrics Optimization for Synthetic Data Generationcited by 6
  14. 2026GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluationcited by 5
  15. 2026Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidencecited by 2

Built on by

Not cited yet.