# Questioning Biases in Case Judgment Summaries: Legal Datasets or Large Language Models?

arXiv: https://arxiv.org/abs/2312.00554 · HTML: https://mlsanity.com/paper/2312.00554/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 19 references that are in the map.
- First posted on arXiv 1 Dec 2023.

## Builds on (most cited first, 6 of 6)

- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [A Short Survey of Viewing Large Language Models in Legal Aspect](https://mlsanity.com/paper/2303.09136/) (2023)
- [How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?](https://mlsanity.com/paper/2306.01248/) (2023)
- [Nonet at SemEval-2023 Task 6: Methodologies for Legal Evaluation](https://mlsanity.com/paper/2310.11049/) (2023)
- [Identifying biases in legal data: An algorithmic fairness perspective](https://mlsanity.com/paper/2109.09946/) (2021)
- [Gender and Racial Stereotype Detection in Legal Opinion Word Embeddings](https://mlsanity.com/paper/2203.13369/) (2022)

## Built on by (most cited first, 7 of 7)

- [Rethinking Legal Judgement Prediction in a Realistic Scenario in the Era of Large Language Models](https://mlsanity.com/paper/2410.10542/) (2024)
- [Code Generation and Algorithmic Problem Solving Using Llama 3.1 405B](https://mlsanity.com/paper/2409.19027/) (2024)
- [Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey](https://mlsanity.com/paper/2404.00990/) (2024)
- [MIRROR: A Novel Approach for the Automated Evaluation of Open-Ended Question Generation](https://mlsanity.com/paper/2410.12893/) (2024)
- [Prompt Engineering Using GPT for Word-Level Code-Mixed Language Identification in Low-Resource Dravidian Languages](https://mlsanity.com/paper/2411.04025/) (2024)
- [Cancer-Answer: Empowering Cancer Care with Advanced Large Language Models](https://mlsanity.com/paper/2411.06946/) (2024)
- [CryptoLLM: Unleashing the Power of Prompted LLMs for SmartQnA and Classification of Crypto Posts](https://mlsanity.com/paper/2411.07917/) (2024)

## Abstract (arXiv)

The evolution of legal datasets and the advent of large language models (LLMs) have significantly transformed the legal field, particularly in the generation of case judgment summaries. However, a critical concern arises regarding the potential biases embedded within these summaries. This study scrutinizes the biases present in case judgment summaries produced by legal datasets and large language models. The research aims to analyze the impact of biases on legal decision making. By interrogating the accuracy, fairness, and implications of biases in these summaries, this study contributes to a better understanding of the role of technology in legal contexts and the implications for justice systems worldwide. In this study, we investigate biases wrt Gender-related keywords, Race-related keywords, Keywords related to crime against women, Country names and religious keywords. The study shows interesting evidences of biases in the outputs generated by the large language models and pre-trained abstractive summarization models. The reasoning behind these biases needs further studies.