# Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset

arXiv: https://arxiv.org/abs/2402.17013 · HTML: https://mlsanity.com/paper/2402.17013/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 73 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Builds on (most cited first, 30 of 36)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [A Survey of the State of Explainable AI for Natural Language Processing](https://mlsanity.com/paper/2010.00711/) (2020)
- [ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation](https://mlsanity.com/paper/2105.13562/) (2021)
- [Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases](https://mlsanity.com/paper/2103.13084/) (2021)
- [Training individually fair ML models with Sensitive Subspace Robustness](https://mlsanity.com/paper/1907.00020/) (2019)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [Learning to Predict Charges for Criminal Cases with Legal Basis](https://mlsanity.com/paper/1707.09168/) (2017)
- [Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://mlsanity.com/paper/2110.00806/) (2021)
- [FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing](https://mlsanity.com/paper/2203.07228/) (2022)
- [Average Individual Fairness: Algorithms, Generalization and Experiments](https://mlsanity.com/paper/1905.10607/) (2019)
- [An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification](https://mlsanity.com/paper/2210.05529/) (2022)
- [Exploring the Use of Text Classification in the Legal Domain](https://mlsanity.com/paper/1710.09306/) (2017)
- [Predicting the Law Area and Decisions of French Supreme Court Cases](https://mlsanity.com/paper/1708.01681/) (2017)
- [On the Ethical Limits of Natural Language Processing on Legal Text](https://mlsanity.com/paper/2105.02751/) (2021)
- [Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts](https://mlsanity.com/paper/2210.13836/) (2022)
- [ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US](https://mlsanity.com/paper/2211.00582/) (2022)
- [An Empirical Study on Cross-X Transfer for Legal Judgment Prediction](https://mlsanity.com/paper/2209.12325/) (2022)
- [One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support](https://mlsanity.com/paper/2306.09237/) (2023)

## Built on by (most cited first, 7 of 7)

- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Incorporating Precedents for Legal Judgement Prediction on European Court of Human Rights Cases](https://mlsanity.com/paper/2409.18644/) (2024)
- [AnnoCaseLaw: A Richly-Annotated Dataset For Benchmarking Explainable Legal Judgment Prediction](https://mlsanity.com/paper/2503.00128/) (2025)
- [Gender Bias in Explainability: Investigating Performance Disparity in Post-hoc Methods](https://mlsanity.com/paper/2505.01198/) (2025)
- [The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Cases](https://mlsanity.com/paper/2409.18645/) (2024)
- [Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learning](https://mlsanity.com/paper/2606.27069/) (2026)
- [Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?](https://mlsanity.com/paper/2406.10974/) (2024)

## Abstract (arXiv)

The assessment of explainability in Legal Judgement Prediction (LJP) systems is of paramount importance in building trustworthy and transparent systems, particularly considering the reliance of these systems on factors that may lack legal relevance or involve sensitive attributes. This study delves into the realm of explainability and fairness in LJP models, utilizing Swiss Judgement Prediction (SJP), the only available multilingual LJP dataset. We curate a comprehensive collection of rationales that `support' and `oppose' judgement from legal experts for 108 cases in German, French, and Italian. By employing an occlusion-based explainability approach, we evaluate the explainability performance of state-of-the-art monolingual and multilingual BERT-based LJP models, as well as models developed with techniques such as data augmentation and cross-lingual transfer, which demonstrated prediction performance improvement. Notably, our findings reveal that improved prediction performance does not necessarily correspond to enhanced explainability performance, underscoring the significance of evaluating models from an explainability perspective. Additionally, we introduce a novel evaluation framework, Lower Court Insertion (LCI), which allows us to quantify the influence of lower court information on model predictions, exposing current models' biases.