# LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation

arXiv: https://arxiv.org/abs/2605.19815 · HTML: https://mlsanity.com/paper/2605.19815/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 19 May 2026.

## Builds on (most cited first, 14 of 14)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists](https://mlsanity.com/paper/2403.18771/) (2024)
- [A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering](https://mlsanity.com/paper/2005.05257/) (2020)
- [Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts](https://mlsanity.com/paper/2210.13836/) (2022)
- [Extractive Summarization of Legal Decisions using Multi-task Learning and Maximal Marginal Relevance](https://mlsanity.com/paper/2210.12437/) (2022)
- [Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification](https://mlsanity.com/paper/2402.07214/) (2024)
- [Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset](https://mlsanity.com/paper/2503.23899/) (2025)
- [LeCoPCR: Legal Concept-guided Prior Case Retrieval for European Court of Human Rights cases](https://mlsanity.com/paper/2501.14114/) (2025)
- [From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLP](https://mlsanity.com/paper/2510.12817/) (2025)

## Abstract (arXiv)

Legal proposition generation is central to legal reasoning and doctrinal scholarship, yet remain under-examined in Legal NLP. This paper investigates the automatic generation and evaluation of legal propositions from decisions of the Court of Justice of the European Union using large language models (LLMs). We introduce LP-Eval, a three-step evaluation rubric co-designed with legal experts that decomposes legal proposition quality into formal validity and substantive dimensions. Using this rubric, we release a dataset of two experts' annotations for 100 LLM-generated legal propositions. Our results show that LLMs can generate predominantly well-formed and high-quality propositions, while expert evaluations reveal higher quality for propositions derived from well established cases than from recent ones. We further examine LLMs as evaluators and find that rubric-guided LLM judgments align more closely with expert assessments than direct overall scoring, but remain insensitive to finer-grained distinctions captured by human experts.