mlsanityData to 13 Oct 2026

LawBench: Benchmarking Legal Knowledge of Large Language Models

arXiv 2309.16289 ↗First posted 28 Sep 2023cs.CL cs.AI cs.LGcited by 116

Not on the map. Why?

Cited by 23 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have demonstrated strong capabilities in various aspects. However, when applying them to the highly specialized, safe-critical legal domain, it is unclear how much legal knowledge they possess and whether they can reliably perform legal-related tasks. To address this gap, we propose a comprehensive evaluation benchmark LawBench. LawBench has been meticulously crafted to have precise assessment of the LLMs' legal capabilities from three cognitive levels: (1) Legal knowledge memorization: whether LLMs can memorize needed legal concepts, articles and facts; (2) Legal knowledge understanding: whether LLMs can comprehend entities, events and relationships within legal text; (3) Legal knowledge applying: whether LLMs can properly utilize their legal knowledge and make necessary reasoning steps to solve realistic legal tasks. LawBench contains 20 diverse tasks covering 5 task types: single-label classification (SLC), multi-label classification (MLC), regression, extraction and generation. We perform extensive evaluations of 51 LLMs on LawBench, including 20 multilingual LLMs, 22 Chinese-oriented LLMs and 9 legal specific LLMs. The results show that GPT-4 remains the best-performing LLM in the legal domain, surpassing the others by a significant margin. While fine-tuning LLMs on legal specific text brings certain improvements, we are still a long way from obtaining usable and reliable LLMs in legal tasks. All data, model predictions and evaluation code are released in https://github.com/open-compass/LawBench/. We hope this benchmark provides in-depth understanding of the LLMs' domain-specified capabilities and speed up the development of LLMs in the legal domain.

Key facts. Cited by 116 papers in the map (data through 13 Oct 2026). Builds on 63 of its 87 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

63 of 87 references.

References · 63

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  7. 2020Scaling Laws for Neural Language Modelscited by 6,494
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  10. 2017Deep reinforcement learning from human preferencescited by 3,871
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,118
38 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,985
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,785
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  4. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,029
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 751
  7. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 743
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 705
  9. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 696
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 617
  11. 2023Active Retrieval Augmented Generationcited by 510
  12. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 439
  13. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 337
  14. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 333
  15. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 304
  16. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 303
  17. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 297
  18. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  19. 2022Transformer Memory as a Differentiable Search Indexcited by 243
  20. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 183
  21. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 166
  22. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  23. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 119
  24. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 115
  25. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 115
  26. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 114
  27. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  28. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 94
  29. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 86
  30. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  31. 2022GPT Takes the Bar Examcited by 65
  32. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  33. 2023Lawyer LLaMA Technical Reportcited by 54
  34. 2023FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Modelscited by 52
  35. 2022A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challengescited by 50
  36. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  37. 2021A Statutory Article Retrieval Dataset in Frenchcited by 40
  38. 2022LEVEN: A Large-Scale Chinese Legal Event Detection Datasetcited by 36

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 27; 2025: 42; 2026: 46.

116 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 473
  2. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 205
  3. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 149
  4. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 62
  5. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 59
  6. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Examscited by 53
  7. 2026Meta Context Engineering via Agentic Skill Evolutioncited by 45
  8. 2024LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Modelscited by 37
  9. 2024LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Modelcited by 34
  10. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 27
  11. 2024Octopus: On-device language model for function calling of software APIscited by 26
  12. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 22
38 more
  1. 2026SIA: Self Improving AI with Harness & Weight Updatescited by 19
  2. 2025PIKE-RAG: sPecIalized KnowledgE and Rationale Augmented Generationcited by 18
  3. 2024IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoningcited by 16
  4. 2024ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Modelscited by 15
  5. 2026MemoHarness: Agent Harnesses That Learn from Experiencecited by 15
  6. 2024Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlationscited by 14
  7. 2025Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environmentscited by 13
  8. 2024CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Modelscited by 12
  9. 2024Do We Need Domain-Specific Embedding Models? An Empirical Investigationcited by 12
  10. 2024BianCang: A Traditional Chinese Medicine Large Language Modelcited by 11
  11. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 11
  12. 2025L-MARS: Legal Multi-Agent System with Agentic Search and Citation-Faithfulness Auditcited by 11
  13. 2025Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Modelscited by 10
  14. 2025Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoningcited by 9
  15. 2025ContractEval: Benchmarking LLMs for Clause-Level Legal Risk Identification in Commercial Contractscited by 9
  16. 2025LawGPT: Knowledge-Guided Data Generation and Its Application to Legal LLMcited by 8
  17. 2026TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Explorationcited by 7
  18. 2024Evaluating Large Language Models for Public Health Classification and Extraction Taskscited by 6
  19. 2024Beyond Guilt: Legal Judgment Prediction with Trichotomous Reasoningcited by 6
  20. 2025GLARE: Agentic Reasoning for Legal Judgment Predictioncited by 6
  21. 2025AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Fieldcited by 6
  22. 2025Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Huntingcited by 6
  23. 2024InternLM-Law: An Open Source Chinese Legal Large Language Modelcited by 5
  24. 2024Large Language Models in Politics and Democracy: A Comprehensive Surveycited by 5
  25. 2025JUREX-4E: Juridical Expert-Annotated Four-Element Knowledge Base for Legal Reasoningcited by 5
  26. 2025Enterprise Large Language Model Evaluation Benchmarkcited by 5
  27. 2025AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliancecited by 5
  28. 2026LLM Agents in Law: Taxonomy, Applications, and Challengescited by 5
  29. 2026Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selectioncited by 5
  30. 2024Aalap: AI Assistant for Legal & Paralegal Functions in Indiacited by 4
  31. 2024ActionReasoningBench: Reasoning about Actions with and without Ramification Constraintscited by 4
  32. 2024LAR-ECHR: A New Legal Argument Reasoning Task and Dataset for Cases of the European Court of Human Rightscited by 4
  33. 2025DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Responsecited by 4
  34. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 4
  35. 2024IPEval: A Bilingual Intellectual Property Agency Consultation Evaluation Benchmark for Large Language Modelscited by 3
  36. 2025Taxation Perspectives from Large Language Models: A Case Study on Additional Tax Penaltiescited by 3
  37. 2025AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarioscited by 3
  38. 2025Measuring Faithfulness and Abstention: An Automated Pipeline for Evaluating LLM-Generated 3-ply Case-Based Legal Argumentscited by 3

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
  2. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
  3. 2024Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models
  4. 2025Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond
  5. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domain
  6. 2025Evaluating the Limits of Large Language Models in Multilingual Legal Reasoning
  7. 2026LexKairos: Benchmarking Legal Temporal Capabilities in LLMs
  8. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice