# Validate Your Authority: Benchmarking LLMs on Multi-Label Precedent Treatment Classification

arXiv: https://arxiv.org/abs/2605.17691 · HTML: https://mlsanity.com/paper/2605.17691/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 22 references that are in the map.
- First posted on arXiv 17 May 2026.

## Builds on (most cited first, 5 of 5)

- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformer](https://mlsanity.com/paper/2211.00974/) (2022)
- [To Aggregate or Not to Aggregate. That is the Question: A Case Study on Annotation Subjectivity in Span Prediction](https://mlsanity.com/paper/2408.02257/) (2024)
- [LegalGuardian: A Privacy-Preserving Framework for Secure Integration of Large Language Models in Legal Practice](https://mlsanity.com/paper/2501.10915/) (2025)

## Built on by (most cited first, 1 of 1)

- [LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI](https://mlsanity.com/paper/2606.18021/) (2026)

## Abstract (arXiv)

Automating the classification of negative treatment in legal precedent is a critical yet nuanced NLP task where misclassification carries significant risk. To address the shortcomings of standard accuracy, this paper introduces a more robust evaluation framework. We benchmark modern Large Language Models on a new, expert-annotated dataset of 239 real-world legal citations and propose a novel Average Severity Error metric to better measure the practical impact of classification errors. Our experiments reveal a performance split. Google's Gemini 2.5 Flash achieved the highest accuracy on a high-level classification task (79.1%), while OpenAI's GPT-5-mini was the top performer on the more complex fine-grained schema (67.7%). This work establishes a crucial baseline, provides a new context-rich dataset, and introduces an evaluation metric tailored to the demands of this complex legal reasoning task.