mlsanityData to 7 Oct 2026

BLT: Can Large Language Models Handle Basic Legal Text?

arXiv 2311.09693 ↗First posted 16 Nov 2023cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

We find that the best publicly available LLMs like GPT-4 and Claude currently perform poorly on basic legal text handling. This motivates the creation of a benchmark consisting of examples that lawyers and paralegals would expect LLMs to handle zero-shot, such as looking up the text at a line of a witness deposition or at a subsection of a contract. LLMs' poor performance on this benchmark casts into doubt their reliability as-is for legal practice. However, fine-tuning on our training set brings even a small model to near-perfect performance. This benchmark will be useful for fine-tuning LLMs for downstream legal tasks, as well as for tracking LLMs' reliability as-is for basic legal tasks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 17 of its 51 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

17 of 51 references.

References · 17

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  6. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  7. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  8. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  9. 2019Neural Legal Judgment Prediction in Englishcited by 208
  10. 2018CAIL2018: A Large-Scale Legal Dataset for Judgment Predictioncited by 168
  11. 2020How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligencecited by 128
  12. 2022Successive Prompting for Decomposing Complex Questionscited by 94
5 more
  1. 2023Can GPT-3 Perform Statutory Reasoning?cited by 71
  2. 2020A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answeringcited by 61
  3. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergencecited by 42
  4. 2022LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoningcited by 17
  5. 2023OpenAI Cribbed Our Tax Example, But Can GPT-4 Really Do Tax?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 2.

10 papers.

  1. 2024Chain of Logic: Rule-Based Reasoning with Large Language Modelscited by 18
  2. 2025Automatic Legal Writing Evaluation of LLMscited by 11
  3. 2024It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Humancited by 5
  4. 2024Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluationscited by 4
  5. 2025Evaluating LLM-Generated Legal Explanations for Regulatory Compliance in Social Media Influencer Marketingcited by 4
  6. 2025Bye-bye, Bluebook? Automating Legal Drudgery With AI-Augmented Rule Followingcited by 3
  7. 2024Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Studycited by 1
  8. 2025Predicting potentially abusive clauses in Chilean terms of services with natural language processingcited by 1
  9. 2026Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QAcited by 0
  10. 2026Beyond Accuracy and Surface Fluency: Risk-Sensitive Evaluation of LLMs for Legal Clause Generationcited by 0