mlsanityData to 7 Oct 2026

Evaluating Rubric Generation with Interventional Transfer

arXiv 2610.10809 ↗First posted 7 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Instance-specific rubrics are common in AI benchmarks where reliable evaluation requires specific expert knowledge. This approach is difficult to scale, prompting research into the generation of rubrics with large language models (LLMs). However, even when expert rubrics are available as references, it is unclear how to productively evaluate the quality of generated rubrics at scale. In this paper, we introduce a method for the evaluation of rubric generation, which we call Interventional Transfer (IT), based on the idea that two rubrics are similar if they move together when a response is perturbed to pass/fail one of them. In contrast to existing approaches for evaluation of rubric generation, we argue that different forms of interventional transfer can be used to evaluate the utility of generated rubrics for different tasks. For instance, we apply this approach in a case study on HealthBench, where we demonstrate an asymmetry in rubrics generated by Qwen3.8-27B, Deepseek-V4-Flash, and Opus-5, used to evaluate responses from GPT-5.6-Terra. Perturbations that degrade responses according to the generated/expert rubric transfer into lower scores on the corresponding expert/generated rubric, but perturbations that improve on one rubric do not reliably transfer into higher scores on the other. We argue that this finding has implications for the usage of LLM-generated rubrics for performance monitoring and hill-climbing. We contrast our approach with existing approaches for rubric evaluation, which do not surface the same asymmetry that we observe.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 30 references that are in the map. First posted on arXiv 7 Oct 2026.

Builds on

27 of 30 references.

References · 27

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  3. 2023GPTScore: Evaluate as You Desirecited by 357
  4. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  5. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  6. 2025Reinforcement Learning with Rubric Anchorscited by 108
  7. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  8. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  9. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 80
  10. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  11. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  12. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
15 more
  1. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
  2. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 19
  3. 2026Learning to Judge: LLMs Designing and Applying Evaluation Rubricscited by 18
  4. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 13
  5. 2025Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoningcited by 10
  6. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 10
  7. 2026Health-SCORE: Towards Scalable Rubrics for Improving Health-LLMscited by 7
  8. 2026Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judgecited by 7
  9. 2026Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproductioncited by 6
  10. 2026Retrieval-Augmented Agentic Rubric Generation for Reliable Medical Response Evaluationcited by 4
  11. 2026Compared to What? Baselines and Metrics for Counterfactual Promptingcited by 4
  12. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  13. 2026The Hitchhikers Guide to Rubric Quality Understanding and Enrichmentcited by 3
  14. 2026Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agentscited by 2
  15. 2026GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluationcited by 2

Built on by

Not cited yet.