# Equality before the Law: Legal Judgment Consistency Analysis for Fairness

arXiv: https://arxiv.org/abs/2103.13868 · HTML: https://mlsanity.com/paper/2103.13868/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 25 Mar 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence](https://mlsanity.com/paper/2004.12158/) (2020)
- [JEC-QA: A Legal-Domain Question Answering Dataset](https://mlsanity.com/paper/1911.12011/) (2019)
- [A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices](https://mlsanity.com/paper/1807.00787/) (2018)
- [Interpretable Charge Predictions for Criminal Cases: Learning to Generate Court Views from Fact Descriptions](https://mlsanity.com/paper/1802.08504/) (2018)
- [Human Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction](https://mlsanity.com/paper/1802.09548/) (2018)
- [Charge-Based Prison Term Prediction with Deep Gating Network](https://mlsanity.com/paper/1908.11521/) (2019)
- [CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain](https://mlsanity.com/paper/1911.08962/) (2019)
- [SECaps: A Sequence Enhanced Capsule Model for Charge Prediction](https://mlsanity.com/paper/1810.04465/) (2018)

## Built on by (most cited first, 11 of 11)

- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://mlsanity.com/paper/2110.00806/) (2021)
- [FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing](https://mlsanity.com/paper/2203.07228/) (2022)
- [LEVEN: A Large-Scale Chinese Legal Event Detection Dataset](https://mlsanity.com/paper/2203.08556/) (2022)
- [Leveraging Task Dependency and Contrastive Learning for Case Outcome Classification on European Court of Human Rights Cases](https://mlsanity.com/paper/2302.00768/) (2023)
- [Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset](https://mlsanity.com/paper/2402.17013/) (2024)
- [LawPal : A Retrieval Augmented Generation Based System for Enhanced Legal Accessibility in India](https://mlsanity.com/paper/2502.16573/) (2025)
- [When Fairness Isn't Statistical: The Limits of Machine Learning in Evaluating Legal Reasoning](https://mlsanity.com/paper/2506.03913/) (2025)
- [The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Cases](https://mlsanity.com/paper/2409.18645/) (2024)
- [The Judge Variable: Challenging Judge-Agnostic Legal Judgment Prediction](https://mlsanity.com/paper/2507.13732/) (2025)
- [Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?](https://mlsanity.com/paper/2406.10974/) (2024)

## Abstract (arXiv)

In a legal system, judgment consistency is regarded as one of the most important manifestations of fairness. However, due to the complexity of factual elements that impact sentencing in real-world scenarios, few works have been done on quantitatively measuring judgment consistency towards real-world data. In this paper, we propose an evaluation metric for judgment inconsistency, Legal Inconsistency Coefficient (LInCo), which aims to evaluate inconsistency between data groups divided by specific features (e.g., gender, region, race). We propose to simulate judges from different groups with legal judgment prediction (LJP) models and measure the judicial inconsistency with the disagreement of the judgment results given by LJP models trained on different groups. Experimental results on the synthetic data verify the effectiveness of LInCo. We further employ LInCo to explore the inconsistency in real cases and come to the following observations: (1) Both regional and gender inconsistency exist in the legal system, but gender inconsistency is much less than regional inconsistency; (2) The level of regional inconsistency varies little across different time periods; (3) In general, judicial inconsistency is negatively correlated with the severity of the criminal charges. Besides, we use LInCo to evaluate the performance of several de-bias methods, such as adversarial learning, and find that these mechanisms can effectively help LJP models to avoid suffering from data bias.