mlsanityData to 7 Oct 2026

MultiLegalPile: A 689GB Multilingual Legal Corpus

arXiv 2306.02069 ↗First posted 3 Jun 2023cs.CL cs.AI cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

Large, high-quality datasets are crucial for training Large Language Models (LLMs). However, so far, there are few datasets available for specialized critical domains such as law and the available ones are often only for the English language. We curate and release MultiLegalPile, a 689GB corpus in 24 languages from 17 jurisdictions. The MultiLegalPile corpus, which includes diverse legal data sources with varying licenses, allows for pretraining NLP models under fair use, with more permissive licenses for the Eurlex Resources and Legal mC4 subsets. We pretrain two RoBERTa models and one Longformer multilingually, and 24 monolingual models on each of the language-specific subsets and evaluate them on LEXTREME. Additionally, we evaluate the English and multilingual models on LexGLUE. Our multilingual models set a new SotA on LEXTREME and our English models on LexGLUE. We release the dataset, the trained models, and all of the code under the most open possible licenses.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 51 of its 79 references that are in the map. First posted on arXiv 3 Jun 2023.

Builds on

51 of 79 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2016Pointer Sentinel Mixture Modelscited by 2,992
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  9. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  10. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  13. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  14. 2022Galactica: A Large Language Model for Sciencecited by 736
  15. 2019Defending Against Neural Fake Newscited by 553
  16. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  17. 2023Scaling Data-Constrained Language Modelscited by 391
  18. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  19. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  20. 2019Neural Legal Judgment Prediction in Englishcited by 208
  21. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  22. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  23. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  24. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  25. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  26. 2023Foundation Models and Fair Usecited by 112
  27. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
  28. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  29. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  30. 2018CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Servicecited by 72
  31. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  32. 2021Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmarkcited by 60
  33. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  34. 2021On the Ethical Limits of Natural Language Processing on Legal Textcited by 26
  35. 2022An Empirical Study on Cross-X Transfer for Legal Judgment Predictioncited by 16
  36. 2021Multi-granular Legal Topic Classification on Greek Legislationcited by 15
  37. 2022BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?cited by 13
  38. 2021Spanish Legalese Language Model and Corporacited by 11

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 20; 2025: 14; 2026: 18.

59 papers.

Sort:

By area:

  1. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  2. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
  3. 2024Non-Determinism of "Deterministic" LLM Settingscited by 84
  4. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  5. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  6. 2023One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Supportcited by 15
  7. 2024BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Modelscited by 15
  8. 2023The Cambridge Law Corpus: A Dataset for Legal AI Researchcited by 14
  9. 2024Can Large Language Models Grasp Legal Theories? Enhance Legal Reasoning with Insights from Multi-Agent Collaborationcited by 14
  10. 2024LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Textcited by 12
  11. 2024Pruning as a Domain-specific LLM Extractorcited by 12
  12. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 12
38 more
  1. 2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12
  2. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  3. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  4. 2024LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domaincited by 10
  5. 2024LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Datasetcited by 8
  6. 2023Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Modelscited by 7
  7. 2024LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in Englishcited by 7
  8. 2024Exploring the Nexus of Large Language Models and Legal Systems: A Short Surveycited by 6
  9. 2025A Comprehensive Survey on Legal Summarization: Challenges and Future Directionscited by 6
  10. 2025A Survey of Classification Tasks and Approaches for Legal Contractscited by 6
  11. 2025The KL3M Data Project: Copyright-Clean Training Resources for Large Language Modelscited by 5
  12. 2024Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Modelscited by 4
  13. 2024All-in-One Tuning and Structural Pruning for Domain-Specific LLMscited by 4
  14. 2025Compressing Language Models for Specialized Domainscited by 4
  15. 2025Optimal Transport Depth Up-Scalingcited by 4
  16. 2023Resolving Legalese: A Multilingual Exploration of Negation Scope Resolution in Legal Documentscited by 3
  17. 2023Automatic Anonymization of Swiss Federal Supreme Court Rulingscited by 3
  18. 2025WisWheat: A Three-Tiered Vision-Language Dataset for Wheat Managementcited by 3
  19. 2026OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demandcited by 3
  20. 2025LegalBench.PT: A Benchmark for Portuguese Lawcited by 2
  21. 2025Identifying Legal Holdings with LLMs: A Systematic Study of Performance, Scale, and Memorizationcited by 2
  22. 2025Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classificationcited by 2
  23. 2026ComplianceNLP: Knowledge-Graph-Augmented RAG for Multi-Framework Regulatory Gap Detectioncited by 2
  24. 2025EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruningcited by 1
  25. 2026Aggregation Queries over Unstructured Text: Benchmark and Agentic Methodcited by 1
  26. 2026LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrievalcited by 1
  27. 2026SinhaLegal: A Benchmark Corpus for Information Extraction and Analysis in Sinhala Legislative Textscited by 1
  28. 2026Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Taskscited by 1
  29. 2026UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoningcited by 1
  30. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditionscited by 1
  31. 2023Legal NLP Meets MiCAR: Advancing the Analysis of Crypto White Paperscited by 0
  32. 2024The Russian Legislative Corpuscited by 0
  33. 2024Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?cited by 0
  34. 2025MEL: Legal Spanish Language Modelcited by 0
  35. 2025Health Insurance Coverage Rule Interpretation Corpus: Law, Policy, and Medical Guidance for Health Insurance Coverage Understandingcited by 0
  36. 2025LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documentscited by 0
  37. 2025Scale-free Characteristics of Multilingual Legal Texts and the Limitations of LLMscited by 0
  38. 2026MAP-Law: Coverage-Driven Retrieval Control for Multi-Turn Legal Consultationcited by 0

50 most cited of 59 All 59 (JSON)