mlsanityData to 7 Oct 2026

FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing

arXiv 2203.07228 ↗First posted 14 Mar 2022cs.CLcited by 48

Not on the map. Why?

Abstract (arXiv)

We present a benchmark suite of four datasets for evaluating the fairness of pre-trained language models and the techniques used to fine-tune them for downstream tasks. Our benchmarks cover four jurisdictions (European Council, USA, Switzerland, and China), five languages (English, German, French, Italian and Chinese) and fairness across five attributes (gender, age, region, language, and legal area). In our experiments, we evaluate pre-trained language models using several group-robust fine-tuning techniques and show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparities. Furthermore, we provide a quantitative and qualitative analysis of our results, highlighting open challenges in the development of robustness methods in legal NLP.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 32 of its 66 references that are in the map. First posted on arXiv 14 Mar 2022.

Builds on

32 of 66 references.

References · 32

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  6. 2019Invariant Risk Minimizationcited by 1,832
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  10. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  11. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  12. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 443
20 more
  1. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  2. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  3. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  4. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  5. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  6. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  7. 2018Empirical Risk Minimization under Fairness Constraintscited by 226
  8. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  9. 2018CAIL2018: A Large-Scale Legal Dataset for Judgment Predictioncited by 168
  10. 2018Adversarial Removal of Demographic Attributes from Text Datacited by 166
  11. 2020How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligencecited by 128
  12. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  13. 2020We Need to Talk About Random Splitscited by 70
  14. 2021Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmarkcited by 60
  15. 2019Discrimination in the Age of Algorithmscited by 51
  16. 2020Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis?cited by 27
  17. 2021On the Ethical Limits of Natural Language Processing on Legal Textcited by 26
  18. 2021Measuring Law Over Time: A Network Analytical Framework with an Application to Statutes and Regulations in the United States and Germanycited by 17
  19. 2021Equality before the Law: Legal Judgment Consistency Analysis for Fairnesscited by 11
  20. 2020Measuring justice in machine learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 15; 2024: 13; 2025: 7; 2026: 7.

48 papers.

Sort:

By area:

  1. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  4. 2023MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRAcited by 80
  5. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  6. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  7. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  8. 2023A Survey on Knowledge Editing of Neural Networkscited by 34
  9. 2022On Learning Fairness and Accuracy on Multiple Subgroupscited by 31
  10. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  11. 2023On the Independence of Association Bias and Empirical Fairness in Language Modelscited by 29
  12. 2023Revisiting non-English Text Simplification: A Unified Multilingual Benchmarkcited by 27
36 more
  1. 2022Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Expertscited by 24
  2. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  3. 2023One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Supportcited by 15
  4. 2023Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Casescited by 14
  5. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  6. 2023Bringing order into the realm of Transformer-based language models for artificial intelligence and lawcited by 13
  7. 2023Leveraging Task Dependency and Contrastive Learning for Case Outcome Classification on European Court of Human Rights Casescited by 12
  8. 2023Being Right for Whose Right Reasons?cited by 12
  9. 2024Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classificationcited by 9
  10. 2023On the Interplay between Fairness and Explainabilitycited by 7
  11. 2024Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Datasetcited by 7
  12. 2024LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in Englishcited by 7
  13. 2024Incorporating Precedents for Legal Judgement Prediction on European Court of Human Rights Casescited by 4
  14. 2024Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Modelscited by 4
  15. 2022Systematic Evaluation of Predictive Fairnesscited by 3
  16. 2024Are Compressed Language Models Less Subgroup Robust?cited by 3
  17. 2022Knowledge is Power: Understanding Causality Makes Legal judgment Prediction Models More Generalizable and Robustcited by 2
  18. 2024FairFlow: An Automated Approach to Model-based Counterfactual Data Augmentation For NLPcited by 2
  19. 2025LegalBench.PT: A Benchmark for Portuguese Lawcited by 2
  20. 2025Computational Law: Datasets, Benchmarks, and Ontologiescited by 2
  21. 2025LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Textcited by 2
  22. 2023Empowering Refugee Claimants and their Lawyers: Using Machine Learning to Examine Decision-Making in Refugee Lawcited by 1
  23. 2024The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Casescited by 1
  24. 2025LegalViz: Legal Text Visualization by Text To Diagram Generationcited by 1
  25. 2025JustEva: A Toolkit to Evaluate LLM Fairness in Legal Knowledge Inferencecited by 1
  26. 2025Does Weak-to-strong Generalization Happen under Spurious Correlations?cited by 1
  27. 2026AppellateGen: A Benchmark for Appellate Legal Judgment Generationcited by 1
  28. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directionscited by 1
  29. 2026Reversible Lifelong Model Editing via Semantic Routing-Based LoRAcited by 1
  30. 2026KrishokChat: A Provenance-Traceable Multi-Task Bengali Agricultural Benchmark with Safety-Critical Chemical Advisorycited by 1
  31. 2023Do Not Harm Protected Groups in Debiasing Language Representation Modelscited by 0
  32. 2024Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?cited by 0
  33. 2025HALF: Harm-Aware LLM Fairness Evaluation Aligned with Deploymentcited by 0
  34. 2026Temporal Concept Drift in Legal Judgment Prediction: Neural Baselines Across Three Epochs of Ukrainian Court Decisionscited by 0
  35. 2026Calibrated Trust, Not Sharper Prediction: An Empirical Test of Uncertainty Fusioncited by 0
  36. 2026SCM-based Fairness and Faithful Explainability for Legal Document Classificationcited by 0