mlsanityData to 14 Oct 2026

LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

arXiv 2110.00976 ↗First posted 3 Oct 2021cs.CLcited by 310

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Laws and their interpretations, legal arguments and agreements are typically expressed in writing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) technologies can be a valuable tool to support legal practitioners in these endeavors. Their usefulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improvements across multiple tasks.

Key facts. Cited by 310 papers in the map (data through 14 Oct 2026). Builds on 50 of its 118 references that are in the map. First posted on arXiv 3 Oct 2021.

Builds on

50 of 118 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2020Longformer: The Long-Document Transformercited by 3,413
38 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  4. 2020Reformer: The Efficient Transformercited by 1,668
  5. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  6. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  7. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  8. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
  9. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
  10. 2019Neural Legal Judgment Prediction in Englishcited by 209
  11. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  12. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  13. 2020How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligencecited by 129
  14. 2019Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbationscited by 119
  15. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 116
  16. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 116
  17. 2019Large-Scale Multi-Label Text Classification on EU Legislationcited by 104
  18. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 90
  19. 2021ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanationcited by 78
  20. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  21. 2018CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Servicecited by 73
  22. 2019Question Answering for Privacy Policies: Combining Computational and Legal Perspectivescited by 67
  23. 2019Legal Judgment Prediction via Multi-Perspective Bi-Feedback Networkcited by 50
  24. 2020An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labelscited by 44
  25. 2021A Statutory Article Retrieval Dataset in Frenchcited by 40
  26. 2018Interpretable Charge Predictions for Criminal Cases: Learning to Generate Court Views from Fact Descriptionscited by 36
  27. 2020Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matchingcited by 35
  28. 2017Predicting the Law Area and Decisions of French Supreme Court Casescited by 30
  29. 2018LexNLP: Natural language processing and information extraction for legal and regulatory textscited by 28
  30. 2021On the Ethical Limits of Natural Language Processing on Legal Textcited by 26
  31. 2018Obligation and Prohibition Extraction Using Hierarchical RNNscited by 18
  32. 2021Measuring Law Over Time: A Network Analytical Framework with an Application to Statutes and Regulations in the United States and Germanycited by 17
  33. 2017Learning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrixcited by 14
  34. 2021Regulatory Compliance through Doc2Doc Information Retrieval: A case study in EU/UK legislation where text similarity has limitationscited by 13
  35. 2020Complex Societies and the Growth of the Lawcited by 12
  36. 2021Equality before the Law: Legal Judgment Consistency Analysis for Fairnesscited by 11
  37. 2021What About the Precedent: An Information-Theoretic Analysis of Common Lawcited by 11
  38. 2021Neural Contract Element Extraction Revisited: Letters from Sesame Streetcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 31; 2023: 50; 2024: 56; 2025: 69; 2026: 104.

310 papers.

Sort:

By area:

  1. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  2. 2023Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecturecited by 122
  3. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 118
  4. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  5. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  6. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  7. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  8. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  9. 2024CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answeringcited by 72
  10. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  11. 2022GPT Takes the Bar Examcited by 65
  12. 2023MultiLegalPile: A 689GB Multilingual Legal Corpuscited by 60
38 more
  1. 2023Metacognitive Prompting Improves Understanding in Large Language Modelscited by 58
  2. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  3. 2023Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaborationcited by 52
  4. 2022A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challengescited by 50
  5. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 50
  6. 2022FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processingcited by 49
  7. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 49
  8. 2022NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Flycited by 48
  9. 2022Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularitiescited by 47
  10. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  11. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  12. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  13. 2024AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentationcited by 40
  14. 2022Pre-trained Language Models for the Legal Domain: A Case Study on Indian Lawcited by 39
  15. 2022An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classificationcited by 38
  16. 2023Natural Language Processing in the Legal Domaincited by 38
  17. 2024LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Modelscited by 37
  18. 2022LEVEN: A Large-Scale Chinese Legal Event Detection Datasetcited by 36
  19. 2023ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmarkcited by 33
  20. 2023KnowGPT: Knowledge Graph based Prompting for Large Language Modelscited by 31
  21. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  22. 2022Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Expertscited by 24
  23. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  24. 2022Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformercited by 20
  25. 2025LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversationcited by 19
  26. 2024ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Modelscited by 18
  27. 2022LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoningcited by 17
  28. 2022ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the UScited by 17
  29. 2024Multi-word Tokenization for Sequence Compressioncited by 17
  30. 2022On the Role of Negative Precedent in Legal Outcome Predictioncited by 16
  31. 2022An Empirical Study on Cross-X Transfer for Legal Judgment Predictioncited by 16
  32. 2022LSG Attention: Extrapolation of pretrained Transformers to long sequencescited by 16
  33. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14
  34. 2023Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Casescited by 14
  35. 2023From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classificationcited by 14
  36. 2022BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?cited by 13
  37. 2023Bringing order into the realm of Transformer-based language models for artificial intelligence and lawcited by 13
  38. 2025Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environmentscited by 13

50 most cited of 310 All 310 (JSON)