mlsanityData to 14 Oct 2026

Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges

arXiv 2608.12097 ↗First posted 12 Aug 2026cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Rubric-based evaluators commonly treat rubrics as prompt context or flat criteria: they specify what to judge but leave criterion composition implicit, even when natural-language rules state it. We introduce Graph-Structured Rubrics (GSR), which compiles a rubric into a response-independent typed evaluation graph before observing responses. Criterion nodes elicit judgments; transformation, reduction, and gating operators compose them through named ports; and a task-specific output mapping, termed Readout, converts the unique sink into a score or preference. Compilation rejects malformed or type-incompatible graphs. Pointwise evaluation judges rubric dimensions separately before graph aggregation; pairwise evaluation reuses the graph with one judgment for each candidate under every criterion. Under GPT-OSS-120B, GSR improves exact score agreement by 0.62--6.75 percentage points over Prometheus-style scoring on four pointwise datasets and achieves the numerically highest end-to-end pairwise accuracy on two preference benchmarks under native tie and abstention policies.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 33 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

31 of 33 references.

References · 31

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  7. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  9. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  11. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  12. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
19 more
  1. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  2. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  3. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  4. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  5. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  6. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  7. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  8. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 74
  9. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  10. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  11. 2023Calibrating LLM-Based Evaluatorcited by 53
  12. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 39
  13. 2024Are LLM-based Evaluators Confusing NLG Quality Criteria?cited by 36
  14. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
  15. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28
  16. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  17. 2026AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Trackingcited by 9
  18. 2026From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judgescited by 7
  19. 2022Confident AIcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Verify Less, Evolve More: Training Idea-Level Critics for Verification-Efficient ML Evolving Agentscited by 0