mlsanityData to 14 Oct 2026

SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges

arXiv 2610.19992 ↗First posted 14 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts. Such answers are expensive to grade, human graders disagree widely on them, and large language models (LLMs) now offer to grade them at scale, both in teaching and in the benchmarks that test legal AI. This paper systematizes how open-text legal answers can be evaluated. We derive eight desiderata from educational measurement and from the law that governs exams, among them validity, reliability, reasons, credit for defensible alternative solutions, robustness to gaming and human oversight. We then compare eight evaluation paradigms against them, from unstructured human grading to LLM judges with instance-specific rubrics and pairwise comparison. Three findings stand out. A rubric written for the task is the strongest lever for agreement between an LLM judge and human graders, but the reported figures answer different questions and rarely come with the human agreement they must be read against. The central validation gap lies where examination law is most demanding: defensible alternative solutions, follow-on errors, and the examiner's correction of model judgments. And rubric-centred grading is the most auditable family, because it makes the standard, its application and the final decision checkable. From this we derive the trustworthy rubric judge: an instance rubric that credits arguments and alternative paths,evidence checked in code, probes, repeated runs, calibration against independent human grades,logging, and an examiner who reads every script. We state what each safeguard rests on, what remains to be tested, and the uses the evidence supports today.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 61 of its 141 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

61 of 141 references.

References · 61

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  6. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  7. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  8. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  9. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  10. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  11. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  12. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
38 more
  1. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 116
  2. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  3. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  4. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 91
  5. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 84
  6. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  7. 2023Applying Large Language Models and Chain-of-Thought for Automatic Scoringcited by 69
  8. 2025One Token to Fool LLM-as-a-Judgecited by 60
  9. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 59
  10. 2024Can You Trust LLM Judgments? Reliability of LLM-as-a-Judgecited by 56
  11. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
  12. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Examscited by 54
  13. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  14. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 39
  15. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  16. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 38
  17. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 31
  18. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 28
  19. 2025Validating LLM-as-a-Judge Systems under Rating Indeterminacycited by 25
  20. 2022Legal Case Document Summarization: Extractive and Abstractive Methods and their Evaluationcited by 24
  21. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 24
  22. 2025The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMscited by 15
  23. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 15
  24. 2025Automatic Legal Writing Evaluation of LLMscited by 12
  25. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 11
  26. 2025LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluationcited by 10
  27. 2024Unlocking Legal Knowledge: A Multilingual Dataset for Judicial Summarization in Switzerlandcited by 7
  28. 2026From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judgescited by 7
  29. 2025AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliancecited by 6
  30. 2025Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustnesscited by 5
  31. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 5
  32. 2025Exploring LLM Autoscoring Reliability in Large-Scale Writing Assessments Using Generalizability Theorycited by 4
  33. 2025Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revisecited by 4
  34. 2025Artificial-Intelligence Grading Assistance for Handwritten Components of a Calculus Examcited by 3
  35. 2025LLM-as-a-Judge is Bad, Based on AI Attempting the Exam Qualifying for the Member of the Polish National Board of Appealcited by 3
  36. 2025Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilitiescited by 3
  37. 2024On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Educationcited by 2
  38. 2025Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesiscited by 2

50 most cited of 61 All 61 (JSON)

Built on by

Not cited yet.