mlsanityData to 14 Oct 2026

LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoning

arXiv 2209.06120 ↗First posted 13 Sep 2022cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Can foundation models be guided to execute tasks involving legal reasoning? We believe that building a benchmark to answer this question will require sustained collaborative efforts between the computer science and legal communities. To that end, this short paper serves three purposes. First, we describe how IRAC-a framework legal scholars use to distinguish different types of legal reasoning-can guide the construction of a Foundation Model oriented benchmark. Second, we present a seed set of 44 tasks built according to this framework. We discuss initial findings, and highlight directions for new tasks. Finally-inspired by the Open Science movement-we make a call for the legal and computer science communities to join our efforts by contributing new tasks. This work is ongoing, and our progress can be tracked here: https://github.com/HazyResearch/legalbench.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 10 of its 31 references that are in the map. First posted on arXiv 13 Sep 2022.

Builds on

10 of 31 references.

References · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  7. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 249
  8. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
  9. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  10. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 6; 2025: 4; 2026: 1.

17 papers.

Sort:

By area:

  1. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
  2. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  3. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
  4. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  5. 2025On Verifiable Legal Reasoning: A Multi-Agent Framework with Formalized Knowledge Representationscited by 18
  6. 2023One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Supportcited by 16
  7. 2025Explainable Rule Application via Structured Prompting: A Neural-Symbolic Approachcited by 14
  8. 2023Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?cited by 12
  9. 2024Evaluating AI for Law: Bridging the Gap with Open-Source Solutionscited by 11
  10. 2023BLT: Can Large Language Models Handle Basic Legal Text?cited by 10
  11. 2024Evalverse: Unified and Accessible Library for Large Language Model Evaluationcited by 5
  12. 2024Automating IRAC Analysis in Malaysian Contract Law using a Semi-Structured Knowledge Basecited by 5
5 more
  1. 2023Towards Legally Enforceable Hate Speech Detection for Public Forumscited by 4
  2. 2023Embroid: Unsupervised Prediction Smoothing Can Improve Few-Shot Classificationcited by 3
  3. 2026Structured Decomposition for LLM Reasoning: Cross-Domain Validation and Semantic Web Integrationcited by 1
  4. 2025LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documentscited by 0
  5. 2025Does It Tie Out? Towards Autonomous Legal Agents in Venture Capitalcited by 0