# A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

arXiv: https://arxiv.org/abs/2204.04859 · HTML: https://mlsanity.com/paper/2204.04859/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 115 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [Generating Visual Explanations](https://mlsanity.com/paper/1603.08507/) (2016)
- [How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence](https://mlsanity.com/paper/2004.12158/) (2020)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation](https://mlsanity.com/paper/2105.13562/) (2021)
- [Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases](https://mlsanity.com/paper/2103.13084/) (2021)
- [CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Service](https://mlsanity.com/paper/1805.01217/) (2018)
- [Distinguish Confusing Law Articles for Legal Judgment Prediction](https://mlsanity.com/paper/2004.02557/) (2020)
- [Learning to Predict Charges for Criminal Cases with Legal Basis](https://mlsanity.com/paper/1707.09168/) (2017)
- [Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://mlsanity.com/paper/2110.00806/) (2021)
- [Legal Judgment Prediction via Multi-Perspective Bi-Feedback Network](https://mlsanity.com/paper/1905.03969/) (2019)
- [Interpretable Charge Predictions for Criminal Cases: Learning to Generate Court Views from Fact Descriptions](https://mlsanity.com/paper/1802.08504/) (2018)

## Built on by (most cited first, 30 of 50)

- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://mlsanity.com/paper/2309.16289/) (2023)
- [A Short Survey of Viewing Large Language Models in Legal Aspect](https://mlsanity.com/paper/2303.09136/) (2023)
- [Evaluating Superhuman Models with Consistency Checks](https://mlsanity.com/paper/2306.09983/) (2023)
- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [Leveraging Large Language Models for NLG Evaluation: Advances and Challenges](https://mlsanity.com/paper/2401.07103/) (2024)
- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US](https://mlsanity.com/paper/2211.00582/) (2022)
- [PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice](https://mlsanity.com/paper/2601.16669/) (2026)
- [Agents on the Bench: Large Language Model Based Multi Agent Framework for Trustworthy Digital Justice](https://mlsanity.com/paper/2412.18697/) (2024)
- [LAiW: A Chinese Legal Large Language Models Benchmark](https://mlsanity.com/paper/2310.05620/) (2023)
- [Debate-Feedback: A Multi-Agent Framework for Efficient Legal Judgment Prediction](https://mlsanity.com/paper/2504.05358/) (2025)
- [LegalReasoner: Step-wised Verification-Correction for Legal Judgment Reasoning](https://mlsanity.com/paper/2506.07443/) (2025)
- [Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction](https://mlsanity.com/paper/2211.08238/) (2022)
- [PILOT: Legal Case Outcome Prediction with Case Law](https://mlsanity.com/paper/2401.15770/) (2024)
- [Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey](https://mlsanity.com/paper/2404.00990/) (2024)
- [InternLM-Law: An Open Source Chinese Legal Large Language Model](https://mlsanity.com/paper/2406.14887/) (2024)
- [Legal Fact Prediction: The Missing Piece in Legal Judgment Prediction](https://mlsanity.com/paper/2409.07055/) (2024)
- [CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions](https://mlsanity.com/paper/2501.00097/) (2024)
- [Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics](https://mlsanity.com/paper/2512.01020/) (2025)
- [Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documents](https://mlsanity.com/paper/2403.06872/) (2024)
- [General Information Metrics for Improving AI Model Training Efficiency](https://mlsanity.com/paper/2501.02004/) (2025)
- [From Graph to Word Bag: Introducing Domain Knowledge to Confusing Charge Prediction](https://mlsanity.com/paper/2403.04369/) (2024)
- [AnnoCaseLaw: A Richly-Annotated Dataset For Benchmarking Explainable Legal Judgment Prediction](https://mlsanity.com/paper/2503.00128/) (2025)
- [Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedents](https://mlsanity.com/paper/2505.14104/) (2025)
- [AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios](https://mlsanity.com/paper/2505.16514/) (2025)
- [When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance](https://mlsanity.com/paper/2507.07748/) (2025)
- [Knowledge is Power: Understanding Causality Makes Legal judgment Prediction Models More Generalizable and Robust](https://mlsanity.com/paper/2211.03046/) (2022)
- [Quebec Automobile Insurance Question-Answering With Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.09623/) (2024)
- [Can Large Language Models Predict the Outcome of Judicial Decisions?](https://mlsanity.com/paper/2501.09768/) (2025)
- [Computational Law: Datasets, Benchmarks, and Ontologies](https://mlsanity.com/paper/2503.04305/) (2025)

## Abstract (arXiv)

Legal judgment prediction (LJP) applies Natural Language Processing (NLP) techniques to predict judgment results based on fact descriptions automatically. Recently, large-scale public datasets and advances in NLP research have led to increasing interest in LJP. Despite a clear gap between machine and human performance, impressive results have been achieved in various benchmark datasets. In this paper, to address the current lack of comprehensive survey of existing LJP tasks, datasets, models and evaluations, (1) we analyze 31 LJP datasets in 6 languages, present their construction process and define a classification method of LJP with 3 different attributes; (2) we summarize 14 evaluation metrics under four categories for different outputs of LJP tasks; (3) we review 12 legal-domain pretrained models in 3 languages and highlight 3 major research directions for LJP; (4) we show the state-of-art results for 8 representative datasets from different court cases and discuss the open challenges. This paper can provide up-to-date and comprehensive reviews to help readers understand the status of LJP. We hope to facilitate both NLP researchers and legal professionals for further joint efforts in this problem.