mlsanityData to 7 Oct 2026

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

arXiv 2308.11462 ↗First posted 20 Aug 2023cs.CL cs.AI cs.CYcited by 328

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning -- which distinguish between its many forms -- correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables.

Key facts. Cited by 328 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 15 of its 59 references that are in the map. First posted on arXiv 20 Aug 2023.

Builds on

15 of 59 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2018Neural Network Acceptability Judgmentscited by 952
  6. 2021True Few-Shot Learning with Language Modelscited by 343
  7. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  8. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  9. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  10. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
  11. 2018CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Servicecited by 72
  12. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
3 more
  1. 2021LeSICiN: A Heterogeneous Graph-based Approach for Automatic Legal Statute Identification from Indian Legal Documentscited by 23
  2. 2023Generative Interpretationcited by 8
  3. 2023A Computational Analysis of Oral Argument in the Supreme Courtcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 68; 2025: 118; 2026: 131.

328 papers.

Sort:

By area:

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  2. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  3. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  4. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  5. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  6. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  7. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  8. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  9. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  10. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  11. 2024On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agentscited by 92
  12. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
38 more
  1. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  2. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  3. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
  4. 2024CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answeringcited by 70
  5. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
  6. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  7. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  8. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 63
  9. 2024(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advicecited by 61
  10. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 48
  11. 2024LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Modelscited by 36
  12. 2024Smoothie: Label Free Language Model Routingcited by 31
  13. 2024CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Masterycited by 30
  14. 2025Towards Robust Legal Reasoning: Harnessing Logical LLMs in Lawcited by 30
  15. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  16. 2025MedBrowseComp: Benchmarking Medical Deep Research and Computer Usecited by 28
  17. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  18. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievalscited by 27
  19. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
  20. 2024Better Call GPT, Comparing Large Language Models Against Lawyerscited by 25
  21. 2024Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scorescited by 25
  22. 2024From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agentscited by 25
  23. 2024Causality for Large Language Modelscited by 24
  24. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 22
  25. 2024HyPA-RAG: A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applicationscited by 22
  26. 2024Formal Theorem Proving by Rewarding LLMs to Decompose Proofs Hierarchicallycited by 22
  27. 2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 21
  28. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Datacited by 21
  29. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 21
  30. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 21
  31. 2025RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problemscited by 21
  32. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 21
  33. 2025YourBench: Easy Custom Evaluation Sets for Everyonecited by 20
  34. 2024How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHOcited by 19
  35. 2024Evaluating the Performance of Large Language Models via Debatescited by 19
  36. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  37. 2024FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answeringcited by 19
  38. 2024Chain of Logic: Rule-Based Reasoning with Large Language Modelscited by 18

50 most cited of 328 All 328 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice
  2. 2025Thinking Longer, Not Always Smarter: Evaluating LLM Capabilities in Hierarchical Legal Reasoning
  3. 2023One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
  4. 2026BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
  5. 2026BenGER Platform: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks
  6. 2025Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond
  7. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Exams
  8. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domain