mlsanityData to 7 Oct 2026

LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain

arXiv 2301.13126 ↗First posted 30 Jan 2023cs.CL cs.AI cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

Lately, propelled by the phenomenal advances around the transformer architecture, the legal NLP field has enjoyed spectacular growth. To measure progress, well curated and challenging benchmarks are crucial. However, most benchmarks are English only and in legal NLP specifically there is no multilingual benchmark available yet. Additionally, many benchmarks are saturated, with the best models clearly outperforming the best humans and achieving near perfect scores. We survey the legal NLP literature and select 11 datasets covering 24 languages, creating LEXTREME. To provide a fair comparison, we propose two aggregate scores, one based on the datasets and one on the languages. The best baseline (XLM-R large) achieves both a dataset aggregate score a language aggregate score of 61.3. This indicates that LEXTREME is still very challenging and leaves ample room for improvement. To make it easy for researchers and practitioners to use, we release LEXTREME on huggingface together with all the code required to evaluate models and a public Weights and Biases project with all the runs.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 60 of its 112 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

60 of 112 references.

References · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
38 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  3. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  4. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  5. 2019CamemBERT: a Tasty French Language Modelcited by 368
  6. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  7. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  8. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  9. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  10. 2019Evaluating Protein Transfer Learning with TAPEcited by 229
  11. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  12. 2020German's Next Language Modelcited by 144
  13. 2020How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligencecited by 128
  14. 2019Multilingual is not enough: BERT for Finnishcited by 118
  15. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 115
  16. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  17. 2019BERTje: A Dutch BERT Modelcited by 109
  18. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  19. 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 99
  20. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
  21. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  22. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  23. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  24. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  25. 2021Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmarkcited by 60
  26. 2022Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularitiescited by 47
  27. 2020The birth of Romanian BERTcited by 44
  28. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  29. 2020GREEK-BERT: The Greeks visiting Sesame Streetcited by 37
  30. 2020Playing with Words at the National Library of Sweden -- Making a Swedish BERTcited by 35
  31. 2022BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Samplingcited by 30
  32. 2021On the Ethical Limits of Natural Language Processing on Legal Textcited by 26
  33. 2021Czert -- Czech BERT-like Model for Language Representationcited by 22
  34. 2022LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoningcited by 17
  35. 2022ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the UScited by 17
  36. 2020EstBERT: A Pretrained Language-Specific BERT for Estoniancited by 16
  37. 2022An Empirical Study on Cross-X Transfer for Legal Judgment Predictioncited by 16
  38. 2021Multi-granular Legal Topic Classification on Greek Legislationcited by 15

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 19; 2025: 14; 2026: 27.

68 papers.

Sort:

By area:

  1. 2023MultiLegalPile: A 689GB Multilingual Legal Corpuscited by 59
  2. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  3. 2023Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluationcited by 52
  4. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  5. 2024AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentationcited by 39
  6. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  7. 2024ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Modelscited by 18
  8. 2024IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoningcited by 16
  9. 2023Large Language Models are legal but they are not: Making the case for a powerful LegalLLMcited by 12
  10. 2024KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questionscited by 11
  11. 2024LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domaincited by 10
  12. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 10
38 more
  1. 2024CoDa: Constrained Generation based Data Augmentation for Low-Resource NLPcited by 9
  2. 2025Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Modelscited by 9
  3. 2024LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Datasetcited by 8
  4. 2024DOLOMITES: Domain-Specific Long-Form Methodical Taskscited by 8
  5. 2025Just Go Parallel: Improving the Multilingual Capabilities of Large Language Modelscited by 8
  6. 2026MAWARITH: A Dataset and Benchmark for Legal Inheritance Reasoning with LLMscited by 8
  7. 2023MultiLegalSBD: A Multilingual Legal Sentence Boundary Detection Datasetcited by 7
  8. 2024Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Datasetcited by 7
  9. 2024LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in Englishcited by 7
  10. 2025A Survey of Classification Tasks and Approaches for Legal Contractscited by 6
  11. 2024EUROPA: A Legal Multilingual Keyphrase Generation Datasetcited by 5
  12. 2024The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunalcited by 5
  13. 2026Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Studycited by 4
  14. 2026The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penaltycited by 4
  15. 2023Resolving Legalese: A Multilingual Exploration of Negation Scope Resolution in Legal Documentscited by 3
  16. 2023Automatic Anonymization of Swiss Federal Supreme Court Rulingscited by 3
  17. 2024Improving Legal Judgement Prediction in Romanian with Long Text Encoderscited by 3
  18. 2024CPSDBench: A Large Language Model Evaluation Benchmark and Baseline for Chinese Public Security Domaincited by 2
  19. 2025LegalBench.PT: A Benchmark for Portuguese Lawcited by 2
  20. 2025Computational Law: Datasets, Benchmarks, and Ontologiescited by 2
  21. 2025JurisTCU: A Brazilian Portuguese Information Retrieval Dataset with Query Relevance Judgmentscited by 2
  22. 2026Temporal Decay of Co-Citation Predictability: A 20-Year Statute Retrieval Benchmark from 396M Ukrainian Court Citationscited by 2
  23. 2026Fine-grained Claim-level RAG Benchmark for Lawcited by 2
  24. 2023Monolingual and Cross-Lingual Knowledge Transfer for Topic Classificationcited by 1
  25. 2024Assessing In-context Learning and Fine-tuning for Topic Classification of German Web Datacited by 1
  26. 2024From Citations to Criticality: Predicting Legal Decision Influence in the Multilingual Swiss Jurisprudencecited by 1
  27. 2025Predicting potentially abusive clauses in Chilean terms of services with natural language processingcited by 1
  28. 2025LegalViz: Legal Text Visualization by Text To Diagram Generationcited by 1
  29. 2025Uncovering inequalities in new knowledge learning by large language models across different languagescited by 1
  30. 2025LaajMeter: A Framework for LaaJ Evaluationcited by 1
  31. 2025CLaw: Benchmarking Chinese Legal Knowledge in Large Language Models - A Fine-grained Corpus and Reasoning Analysiscited by 1
  32. 2025JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledgecited by 1
  33. 2026AppellateGen: A Benchmark for Appellate Legal Judgment Generationcited by 1
  34. 2026Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Lawcited by 1
  35. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directionscited by 1
  36. 2026Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Taskscited by 1
  37. 2026UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoningcited by 1
  38. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditionscited by 1

50 most cited of 68 All 68 (JSON)