mlsanityData to 14 Oct 2026

LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation

arXiv 2605.19815 ↗First posted 19 May 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Legal proposition generation is central to legal reasoning and doctrinal scholarship, yet remain under-examined in Legal NLP. This paper investigates the automatic generation and evaluation of legal propositions from decisions of the Court of Justice of the European Union using large language models (LLMs). We introduce LP-Eval, a three-step evaluation rubric co-designed with legal experts that decomposes legal proposition quality into formal validity and substantive dimensions. Using this rubric, we release a dataset of two experts' annotations for 100 LLM-generated legal propositions. Our results show that LLMs can generate predominantly well-formed and high-quality propositions, while expert evaluations reveal higher quality for propositions derived from well established cases than from recent ones. We further examine LLMs as evaluators and find that rubric-guided LLM judgments align more closely with expert assessments than direct overall scoring, but remain insensitive to finer-grained distinctions captured by human experts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 31 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

14 of 31 references.

References · 14

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2020Extracting Training Data from Large Language Modelscited by 2,189
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  4. 2019Neural Legal Judgment Prediction in Englishcited by 209
  5. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  6. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
  7. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  8. 2020A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answeringcited by 62
  9. 2022Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Expertscited by 24
  10. 2022Extractive Summarization of Legal Decisions using Multi-task Learning and Maximal Marginal Relevancecited by 9
  11. 2024Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classificationcited by 9
  12. 2025Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE datasetcited by 8
2 more
  1. 2025LeCoPCR: Legal Concept-guided Prior Case Retrieval for European Court of Human Rights casescited by 5
  2. 2025From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLPcited by 3

Built on by

Not cited yet.