# Rethinking Legal Judgement Prediction in a Realistic Scenario in the Era of Large Language Models

arXiv: https://arxiv.org/abs/2410.10542 · HTML: https://mlsanity.com/paper/2410.10542/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 12 of its 35 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [Fine-tune BERT for Extractive Summarization](https://mlsanity.com/paper/1903.10318/) (2019)
- [Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://mlsanity.com/paper/2110.00806/) (2021)
- [How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?](https://mlsanity.com/paper/2306.01248/) (2023)
- [Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law](https://mlsanity.com/paper/2209.06049/) (2022)
- [Legal Question-Answering in the Indian Context: Efficacy, Challenges, and Potential of Modern AI Models](https://mlsanity.com/paper/2309.14735/) (2023)
- [Nonet at SemEval-2023 Task 6: Methodologies for Legal Evaluation](https://mlsanity.com/paper/2310.11049/) (2023)
- [Questioning Biases in Case Judgment Summaries: Legal Datasets or Large Language Models?](https://mlsanity.com/paper/2312.00554/) (2023)
- [Artificial Intelligence (AI) in Legal Data Mining](https://mlsanity.com/paper/2405.14707/) (2024)

## Built on by (most cited first, 10 of 10)

- [JuDGE: Benchmarking Judgment Document Generation for Chinese Legal System](https://mlsanity.com/paper/2503.14258/) (2025)
- [Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention](https://mlsanity.com/paper/2509.24393/) (2025)
- [GLARE: Agentic Reasoning for Legal Judgment Prediction](https://mlsanity.com/paper/2508.16383/) (2025)
- [Enhancing Judgment Document Generation via Agentic Legal Information Collection and Rubric-Guided Optimization](https://mlsanity.com/paper/2605.02011/) (2026)
- [Prompt Engineering Using GPT for Word-Level Code-Mixed Language Identification in Low-Resource Dravidian Languages](https://mlsanity.com/paper/2411.04025/) (2024)
- [Cancer-Answer: Empowering Cancer Care with Advanced Large Language Models](https://mlsanity.com/paper/2411.06946/) (2024)
- [CryptoLLM: Unleashing the Power of Prompted LLMs for SmartQnA and Classification of Crypto Posts](https://mlsanity.com/paper/2411.07917/) (2024)
- [LLMs for Legal Subsumption in German Employment Contracts](https://mlsanity.com/paper/2507.01734/) (2025)
- [Judging by Appearances? Auditing and Intervening Vision-Language Models for Bail Prediction](https://mlsanity.com/paper/2510.00088/) (2025)
- [Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data](https://mlsanity.com/paper/2511.20669/) (2025)

## Abstract (arXiv)

This study investigates judgment prediction in a realistic scenario within the context of Indian judgments, utilizing a range of transformer-based models, including InLegalBERT, BERT, and XLNet, alongside LLMs such as Llama-2 and GPT-3.5 Turbo. In this realistic scenario, we simulate how judgments are predicted at the point when a case is presented for a decision in court, using only the information available at that time, such as the facts of the case, statutes, precedents, and arguments. This approach mimics real-world conditions, where decisions must be made without the benefit of hindsight, unlike retrospective analyses often found in previous studies. For transformer models, we experiment with hierarchical transformers and the summarization of judgment facts to optimize input for these models. Our experiments with LLMs reveal that GPT-3.5 Turbo excels in realistic scenarios, demonstrating robust performance in judgment prediction. Furthermore, incorporating additional legal information, such as statutes and precedents, significantly improves the outcome of the prediction task. The LLMs also provide explanations for their predictions. To evaluate the quality of these predictions and explanations, we introduce two human evaluation metrics: Clarity and Linking. Our findings from both automatic and human evaluations indicate that, despite advancements in LLMs, they are yet to achieve expert-level performance in judgment prediction and explanation tasks.