# The Judge Variable: Challenging Judge-Agnostic Legal Judgment Prediction

arXiv: https://arxiv.org/abs/2507.13732 · HTML: https://mlsanity.com/paper/2507.13732/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 58 references that are in the map.
- First posted on arXiv 18 Jul 2025.

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence](https://mlsanity.com/paper/2004.12158/) (2020)
- [ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation](https://mlsanity.com/paper/2105.13562/) (2021)
- [Distinguish Confusing Law Articles for Legal Judgment Prediction](https://mlsanity.com/paper/2004.02557/) (2020)
- [Learning to Predict Charges for Criminal Cases with Legal Basis](https://mlsanity.com/paper/1707.09168/) (2017)
- [Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://mlsanity.com/paper/2110.00806/) (2021)
- [Legal Judgment Prediction via Multi-Perspective Bi-Feedback Network](https://mlsanity.com/paper/1905.03969/) (2019)
- [Exploring the Use of Text Classification in the Legal Domain](https://mlsanity.com/paper/1710.09306/) (2017)
- [Predicting the Law Area and Decisions of French Supreme Court Cases](https://mlsanity.com/paper/1708.01681/) (2017)
- [Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherence](https://mlsanity.com/paper/2502.14905/) (2025)
- [Equality before the Law: Legal Judgment Consistency Analysis for Fairness](https://mlsanity.com/paper/2103.13868/) (2021)
- [Fact-based Court Judgment Prediction](https://mlsanity.com/paper/2311.13350/) (2023)
- [Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification](https://mlsanity.com/paper/2402.07214/) (2024)
- [Predicting Brazilian court decisions](https://mlsanity.com/paper/1905.10348/) (2019)
- [PILOT: Legal Case Outcome Prediction with Case Law](https://mlsanity.com/paper/2401.15770/) (2024)
- [Predicting the Behavior of the Supreme Court of the United States: A General Approach](https://mlsanity.com/paper/1407.6333/) (2014)
- [An External Knowledge Enhanced Multi-label Charge Prediction Approach with Label Number Learning](https://mlsanity.com/paper/1907.02205/) (2019)
- [RLJP: Legal Judgment Prediction via First-Order Logic Rule-enhanced with Large Language Models](https://mlsanity.com/paper/2505.21281/) (2025)
- [A Minimalist Prompt for Zero-Shot Policy Learning](https://mlsanity.com/paper/2405.06063/) (2024)
- [When Fairness Isn't Statistical: The Limits of Machine Learning in Evaluating Legal Reasoning](https://mlsanity.com/paper/2506.03913/) (2025)
- [Crowdsourcing accurately and robustly predicts Supreme Court decisions](https://mlsanity.com/paper/1712.03846/) (2017)

## Built on by (most cited first, 1 of 1)

- [Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learning](https://mlsanity.com/paper/2606.27069/) (2026)

## Abstract (arXiv)

This study examines the role of human judges in legal decision-making by using machine learning to predict child physical custody outcomes in French appellate courts. Building on the legal realism-formalism debate, we test whether individual judges' decision-making patterns significantly influence case outcomes, challenging the assumption that judges are neutral variables that apply the law uniformly. To ensure compliance with French privacy laws, we implement a strict pseudonymization process. Our analysis uses 18,937 living arrangements rulings extracted from 10,306 cases. We compare models trained on individual judges' past rulings (specialist models) with a judge-agnostic model trained on aggregated data (generalist models). The prediction pipeline is a hybrid approach combining large language models (LLMs) for structured feature extraction and ML models for outcome prediction (RF, XGB and SVC). Our results show that specialist models consistently achieve higher predictive accuracy than the general model, with top-performing models reaching F1 scores as high as 92.85%, compared to the generalist model's 82.63% trained on 20x to 100x more samples. Specialist models capture stable individual patterns that are not transferable to other judges. In-Domain and Cross-Domain validity tests provide empirical support for legal realism, demonstrating that judicial identity plays a measurable role in legal outcomes. All data and code used will be made available.