mlsanityData to 14 Oct 2026

Grading Needs a Rubric, Not Intelligence

arXiv 2608.17938 ↗First posted 18 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Small language models can grade open-ended examination answers as reliably as substantially more expensive models when they grade against an explicit rubric. We test this claim as the design principle behind any-to-bench: a frontier model reads source documents once, at ingestion, to extract each question and its rubric; lower-cost models then perform all repeated grading work. We evaluate six cost-efficient model configurations from two model families at three reasoning-effort levels. Each configuration answers 24 open-ended examination questions, and each also grades every answer sheet three times, yielding 3,456 per-question grades. Scores depend overwhelmingly on the answer being graded: answer identity explains 95.6% of score variance, whereas judge identity explains only 0.2%. Raising a writer's reasoning effort moves earned scores by as much as 0.143 of full marks, while raising a judge's reasoning effort moves assigned scores by at most 0.006. Six frontier-tier judges, added as a check, reproduce these scores and are no more reliable as a panel. Two ablations then decompose the rubric on the same questions and answers. Removing its criteria and levels while keeping the official answer changes nothing measurable. Removing the official answer as well collapses reliability (ICC 0.888 to 0.628), inflates scores, and makes judge reasoning effort matter again. The rubric is what decouples grading from judge intelligence, and within the rubric the official answer does nearly all the work. We find no evidence of length preference or same-family preference under rubric-anchored grading.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 9 of its 11 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

9 of 11 references.

References · 9

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  7. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  8. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  9. 2026Machine-Assisted Grading of Nationwide School-Leaving Essay Exams with LLMs and Statistical NLPcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Grade Like a Human: Rethinking Automated Assessment with Large Language Models
  2. 2025Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness
  3. 2025SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models
  4. 2024Evaluating Students' Open-ended Written Responses with LLMs: Using the RAG Framework for GPT-3.5, GPT-4, Claude-3, and Mistral-Large
  5. 2025SedarEval: Automated Evaluation using Self-Adaptive Rubrics
  6. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation
  7. 2025Concept-based Rubrics Improve LLM Formative Assessment and Data Synthesis
  8. 2025SteLLA: A Structured Grading System Using LLMs with RAG