mlsanityData to 14 Oct 2026

LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development

arXiv 2305.07507 ↗First posted 12 May 2023cs.CLcited by 54

Not on the map. Why?

Abstract (arXiv)

In this work, we conduct a detailed analysis on the performance of legal-oriented pre-trained language models (PLMs). We examine the interplay between their original objective, acquired knowledge, and legal language understanding capacities which we define as the upstream, probing, and downstream performance, respectively. We consider not only the models' size but also the pre-training corpora used as important dimensions in our study. To this end, we release a multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to facilitate training and detailed analysis of legal-oriented PLMs. We release two new legal PLMs trained on LeXFiles and evaluate them alongside others on LegalLAMA and LexGLUE. We find that probing performance strongly correlates with upstream performance in related legal topics. On the other hand, downstream performance is mainly driven by the model's size and prior legal knowledge which can be estimated by upstream and probing performance. Based on these findings, we can conclude that both dimensions are important for those seeking the development of domain-specific PLMs.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 32 of its 57 references that are in the map. First posted on arXiv 12 May 2023.

Builds on

32 of 57 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2020Longformer: The Long-Document Transformercited by 3,413
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
20 more
  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2019Language Models as Knowledge Bases?cited by 1,936
  3. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  4. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  5. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  6. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  7. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 148
  8. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  9. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 116
  10. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  11. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  12. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 90
  13. 2021ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanationcited by 78
  14. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  15. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  16. 2022FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processingcited by 49
  17. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  18. 2022Mining Legal Arguments in Court Decisionscited by 35
  19. 2021On the Ethical Limits of Natural Language Processing on Legal Textcited by 26
  20. 2021The Singleton Fallacy: Why Current Critiques of Language Models Miss the Pointcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 29; 2025: 10; 2026: 7.

54 papers.

Sort:

By area:

  1. 2023MultiLegalPile: A 689GB Multilingual Legal Corpuscited by 60
  2. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17
  3. 2023The Cambridge Law Corpus: A Dataset for Legal AI Researchcited by 14
  4. 2023Large Language Models are legal but they are not: Making the case for a powerful LegalLLMcited by 12
  5. 2024LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Textcited by 12
  6. 2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12
  7. 2024LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domaincited by 10
  8. 2024MuLan: A Study of Fact Mutability in Language Modelscited by 10
  9. 2024CoDa: Constrained Generation based Data Augmentation for Low-Resource NLPcited by 9
  10. 2025DivScore: Zero-Shot Detection of LLM-Generated Text in Specialized Domainscited by 9
  11. 2024LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Datasetcited by 8
  12. 2024ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptionscited by 8
38 more
  1. 2024LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in Englishcited by 8
  2. 2024Defining Knowledge: Bridging Epistemology and Large Language Modelscited by 7
  3. 2024Exploring the Nexus of Large Language Models and Legal Systems: A Short Surveycited by 6
  4. 2025A Survey of Classification Tasks and Approaches for Legal Contractscited by 6
  5. 2024EUROPA: A Legal Multilingual Keyphrase Generation Datasetcited by 5
  6. 2024Length-Aware Multi-Kernel Transformer for Long Document Classificationcited by 5
  7. 2024Automating IRAC Analysis in Malaysian Contract Law using a Semi-Structured Knowledge Basecited by 5
  8. 2024The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunalcited by 5
  9. 2024LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Textcited by 5
  10. 2025LRAGE: Legal Retrieval Augmented Generation Evaluation Toolcited by 5
  11. 2025CoLA: Collaborative Low-Rank Adaptationcited by 5
  12. 2025Explainable Compliance Detection with Multi-Hop Natural Language Inference on Assurance Case Structurecited by 5
  13. 2024Summarizing long regulatory documents with a multi-step pipelinecited by 4
  14. 2024Incorporating Precedents for Legal Judgement Prediction on European Court of Human Rights Casescited by 4
  15. 2024Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Modelscited by 4
  16. 2023Retrieval-augmented Multi-label Text Classificationcited by 3
  17. 2025FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Expertscited by 3
  18. 2025Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarkscited by 3
  19. 2024Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristicscited by 2
  20. 2024Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domaincited by 2
  21. 2024NLP at UC Santa Cruz at SemEval-2024 Task 5: Legal Answer Validation using Few-Shot Multi-Choice QAcited by 2
  22. 2025Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classificationcited by 2
  23. 2023Do Language Models Learn about Legal Entity Types during Pretraining?cited by 1
  24. 2023LaCour!: Enabling Research on Argumentation in Hearings of the European Court of Human Rightscited by 1
  25. 2024The Right Model for the Job: An Evaluation of Legal Multi-Label Classification Baselinescited by 1
  26. 2024The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Casescited by 1
  27. 2025LegalViz: Legal Text Visualization by Text To Diagram Generationcited by 1
  28. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directionscited by 1
  29. 2023Legal NLP Meets MiCAR: Advancing the Analysis of Crypto White Paperscited by 0
  30. 2024BayesJudge: Bayesian Kernel Language Modelling with Confidence Uncertainty in Legal Judgment Predictioncited by 0
  31. 2024Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?cited by 0
  32. 2024eagerlearners at SemEval2024 Task 5: The Legal Argument Reasoning Task in Civil Procedurecited by 0
  33. 2024The Large Language Model GreekLegalRoBERTacited by 0
  34. 2024TDDBench: A Benchmark for Training data detectioncited by 0
  35. 2024MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Datasetcited by 0
  36. 2025PL-CA: A Parametric Legal Case Augmentation Frameworkcited by 0
  37. 2026A Two-Stage Architecture for NDA Analysis: LLM-based Segmentation and Transformer-based Clause Classificationcited by 0
  38. 2026LLM-Assisted Causal Structure Disambiguation and Factor Extraction for Legal Judgment Predictioncited by 0

50 most cited of 54 All 54 (JSON)