# From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification

arXiv: https://arxiv.org/abs/2310.11878 · HTML: https://mlsanity.com/paper/2310.11878/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Reframing Human-AI Collaboration for Generating Free-Text Explanations](https://mlsanity.com/paper/2112.08674/) (2021)
- [Stance Detection with Bidirectional Conditional Encoding](https://mlsanity.com/paper/1606.05464/) (2016)
- [ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation](https://mlsanity.com/paper/2105.13562/) (2021)
- [Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases](https://mlsanity.com/paper/2103.13084/) (2021)
- [Mining Legal Arguments in Court Decisions](https://mlsanity.com/paper/2208.06178/) (2022)
- [On the Diversity and Limits of Human Explanations](https://mlsanity.com/paper/2106.11988/) (2021)
- [Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts](https://mlsanity.com/paper/2210.13836/) (2022)
- [On the Role of Negative Precedent in Legal Outcome Prediction](https://mlsanity.com/paper/2208.08225/) (2022)
- [Leveraging Task Dependency and Contrastive Learning for Case Outcome Classification on European Court of Human Rights Cases](https://mlsanity.com/paper/2302.00768/) (2023)
- [Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation](https://mlsanity.com/paper/2304.12443/) (2023)

## Built on by (most cited first, 14 of 14)

- [Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP](https://mlsanity.com/paper/2601.09065/) (2026)
- [Which Demographics do LLMs Default to During Annotation?](https://mlsanity.com/paper/2410.08820/) (2024)
- [Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification](https://mlsanity.com/paper/2402.07214/) (2024)
- [LexAbSumm: Aspect-based Summarization of Legal Decisions](https://mlsanity.com/paper/2404.00594/) (2024)
- [Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset](https://mlsanity.com/paper/2402.17013/) (2024)
- [Towards Explainability in Legal Outcome Prediction Models](https://mlsanity.com/paper/2403.16852/) (2024)
- [Query-driven Relevant Paragraph Extraction from Legal Judgments](https://mlsanity.com/paper/2404.00595/) (2024)
- [Evaluating LLM-Generated Legal Explanations for Regulatory Compliance in Social Media Influencer Marketing](https://mlsanity.com/paper/2510.08111/) (2025)
- [AnnoCaseLaw: A Richly-Annotated Dataset For Benchmarking Explainable Legal Judgment Prediction](https://mlsanity.com/paper/2503.00128/) (2025)
- [From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLP](https://mlsanity.com/paper/2510.12817/) (2025)
- [The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Cases](https://mlsanity.com/paper/2409.18645/) (2024)
- [Quantifying and Predicting Disagreement in Graded Human Ratings](https://mlsanity.com/paper/2605.01168/) (2026)
- [Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?](https://mlsanity.com/paper/2406.10974/) (2024)
- [Hybrid Deep Learning for Legal Text Analysis: Predicting Punishment Durations in Indonesian Court Rulings](https://mlsanity.com/paper/2410.20104/) (2024)

## Abstract (arXiv)

In legal NLP, Case Outcome Classification (COC) must not only be accurate but also trustworthy and explainable. Existing work in explainable COC has been limited to annotations by a single expert. However, it is well-known that lawyers may disagree in their assessment of case facts. We hence collect a novel dataset RAVE: Rationale Variation in ECHR1, which is obtained from two experts in the domain of international human rights law, for whom we observe weak agreement. We study their disagreements and build a two-level task-independent taxonomy, supplemented with COC-specific subcategories. To our knowledge, this is the first work in the legal NLP that focuses on human label variation. We quantitatively assess different taxonomy categories and find that disagreements mainly stem from underspecification of the legal context, which poses challenges given the typically limited granularity and noise in COC metadata. We further assess the explainablility of SOTA COC models on RAVE and observe limited agreement between models and experts. Overall, our case study reveals hitherto underappreciated complexities in creating benchmark datasets in legal NLP that revolve around identifying aspects of a case's facts supposedly relevant to its outcome.