# BayesJudge: Bayesian Kernel Language Modelling with Confidence Uncertainty in Legal Judgment Prediction

arXiv: https://arxiv.org/abs/2404.10481 · HTML: https://mlsanity.com/paper/2404.10481/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 70 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [Approximate Inference Turns Deep Networks into Gaussian Processes](https://mlsanity.com/paper/1906.01930/) (2019)
- [Distinguish Confusing Law Articles for Legal Judgment Prediction](https://mlsanity.com/paper/2004.02557/) (2020)
- [LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development](https://mlsanity.com/paper/2305.07507/) (2023)
- [A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges](https://mlsanity.com/paper/2204.04859/) (2022)
- [Is MC Dropout Bayesian?](https://mlsanity.com/paper/2110.04286/) (2021)
- [Legal Judgment Prediction with Multi-Stage CaseRepresentation Learning in the Real Court Setting](https://mlsanity.com/paper/2107.05192/) (2021)
- [Bayesian Learning for Neural Networks: an algorithmic survey](https://mlsanity.com/paper/2211.11865/) (2022)
- [Transformer Uncertainty Estimation with Hierarchical Stochastic Attention](https://mlsanity.com/paper/2112.13776/) (2021)
- [ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US](https://mlsanity.com/paper/2211.00582/) (2022)
- [On the Role of Negative Precedent in Legal Outcome Prediction](https://mlsanity.com/paper/2208.08225/) (2022)
- [Learning Fine-grained Fact-Article Correspondence in Legal Cases](https://mlsanity.com/paper/2104.10726/) (2021)
- [To BAN or not to BAN: Bayesian Attention Networks for Reliable Hate Speech Detection](https://mlsanity.com/paper/2007.05304/) (2020)
- [SemEval 2023 Task 6: LegalEval - Understanding Legal Texts](https://mlsanity.com/paper/2304.09548/) (2023)
- [Surrogate uncertainty estimation for your time series forecasting black-box: learn when to trust](https://mlsanity.com/paper/2302.02834/) (2023)

## Abstract (arXiv)

Predicting legal judgments with reliable confidence is paramount for responsible legal AI applications. While transformer-based deep neural networks (DNNs) like BERT have demonstrated promise in legal tasks, accurately assessing their prediction confidence remains crucial. We present a novel Bayesian approach called BayesJudge that harnesses the synergy between deep learning and deep Gaussian Processes to quantify uncertainty through Bayesian kernel Monte Carlo dropout. Our method leverages informative priors and flexible data modelling via kernels, surpassing existing methods in both predictive accuracy and confidence estimation as indicated through brier score. Extensive evaluations of public legal datasets showcase our model's superior performance across diverse tasks. We also introduce an optimal solution to automate the scrutiny of unreliable predictions, resulting in a significant increase in the accuracy of the model's predictions by up to 27%. By empowering judges and legal professionals with more reliable information, our work paves the way for trustworthy and transparent legal AI applications that facilitate informed decisions grounded in both knowledge and quantified uncertainty.