# ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US

arXiv: https://arxiv.org/abs/2211.00582 · HTML: https://mlsanity.com/paper/2211.00582/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 63 references that are in the map.
- First posted on arXiv 1 Nov 2022.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021)
- [BillSum: A Corpus for Automatic Summarization of US Legislation](https://mlsanity.com/paper/1910.00523/) (2019)
- [Multivariate Confidence Calibration for Object Detection](https://mlsanity.com/paper/2004.13546/) (2020)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation](https://mlsanity.com/paper/2105.13562/) (2021)
- [Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://mlsanity.com/paper/2110.00806/) (2021)
- [A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges](https://mlsanity.com/paper/2204.04859/) (2022)
- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities](https://mlsanity.com/paper/2206.10883/) (2022)

## Built on by (most cited first, 17 of 17)

- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text](https://mlsanity.com/paper/2402.04335/) (2024)
- [SwiLTra-Bench: The Swiss Legal Translation Benchmark](https://mlsanity.com/paper/2503.01372/) (2025)
- [LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain](https://mlsanity.com/paper/2404.02127/) (2024)
- [Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset](https://mlsanity.com/paper/2402.17013/) (2024)
- [Japanese Tort-case Dataset for Rationale-supported Legal Judgment Prediction](https://mlsanity.com/paper/2312.00480/) (2023)
- [Legal Fact Prediction: The Missing Piece in Legal Judgment Prediction](https://mlsanity.com/paper/2409.07055/) (2024)
- [LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text](https://mlsanity.com/paper/2410.12064/) (2024)
- [VLQA: The First Comprehensive, Large, and High-Quality Vietnamese Dataset for Legal Question Answering](https://mlsanity.com/paper/2507.19995/) (2025)
- [AnnoCaseLaw: A Richly-Annotated Dataset For Benchmarking Explainable Legal Judgment Prediction](https://mlsanity.com/paper/2503.00128/) (2025)
- [ASP2LJ : An Adversarial Self-Play Laywer Augmented Legal Judgment Framework](https://mlsanity.com/paper/2506.18768/) (2025)
- [CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law](https://mlsanity.com/paper/2605.30497/) (2026)
- [From Citations to Criticality: Predicting Legal Decision Influence in the Multilingual Swiss Jurisprudence](https://mlsanity.com/paper/2410.13460/) (2024)
- [BayesJudge: Bayesian Kernel Language Modelling with Confidence Uncertainty in Legal Judgment Prediction](https://mlsanity.com/paper/2404.10481/) (2024)
- [Understanding Benchmark Language Under Weakened Formal Semantics](https://mlsanity.com/paper/2509.17455/) (2025)
- [When Does Defendant Statement Matter? A Study of Bias and Persuasion in LLM-Simulated Jurors](https://mlsanity.com/paper/2609.09887/) (2026)

## Abstract (arXiv)

The research field of Legal Natural Language Processing (NLP) has been very active recently, with Legal Judgment Prediction (LJP) becoming one of the most extensively studied tasks. To date, most publicly released LJP datasets originate from countries with civil law. In this work, we release, for the first time, a challenging LJP dataset focused on class action cases in the US. It is the first dataset in the common law system that focuses on the harder and more realistic task involving the complaints as input instead of the often used facts summary written by the court. Additionally, we study the difficulty of the task by collecting expert human predictions, showing that even human experts can only reach 53% accuracy on this dataset. Our Longformer model clearly outperforms the human baseline (63%), despite only considering the first 2,048 tokens. Furthermore, we perform a detailed error analysis and find that the Longformer model is significantly better calibrated than the human experts. Finally, we publicly release the dataset and the code used for the experiments.