mlsanityData to 14 Oct 2026

Can GPT-3 Perform Statutory Reasoning?

arXiv 2302.06100 ↗First posted 13 Feb 2023cs.CL cs.AIcited by 73

Not on the map. Why?

Abstract (arXiv)

Statutory reasoning is the task of reasoning with facts and statutes, which are rules written in natural language by a legislature. It is a basic legal skill. In this paper we explore the capabilities of the most capable GPT-3 model, text-davinci-003, on an established statutory-reasoning dataset called SARA. We consider a variety of approaches, including dynamic few-shot prompting, chain-of-thought prompting, and zero-shot prompting. While we achieve results with GPT-3 that are better than the previous best published results, we also identify several types of clear errors it makes. We investigate why these errors happen. We discover that GPT-3 has imperfect prior knowledge of the actual U.S. statutes on which SARA is based. More importantly, we create simple synthetic statutes, which GPT-3 is guaranteed not to have seen during training. We find GPT-3 performs poorly at answering straightforward questions about these simple synthetic statutes.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Builds on 17 of its 29 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

17 of 29 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  10. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  11. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  12. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
5 more
  1. 2022GPT Takes the Bar Examcited by 65
  2. 2020A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answeringcited by 62
  3. 2022Legal Prompting: Teaching a Language Model to Think Like a Lawyercited by 60
  4. 2021Factoring Statutory Reasoning as Language Understanding Challengescited by 20
  5. 2021Discovering Explanatory Sentences in Legal Case Decisions Using Pre-trained Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 22; 2024: 31; 2025: 10; 2026: 10.

73 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023Challenges and Applications of Large Language Modelscited by 254
  3. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  4. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
  5. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  6. 2023MindAgent: Emergent Gaming Interactioncited by 88
  7. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  8. 2023A Short Survey of Viewing Large Language Models in Legal Aspectcited by 52
  9. 2023Explaining Legal Concepts with Augmented Large Language Models (GPT-4)cited by 40
  10. 2023Re-Reading Improves Reasoning in Large Language Modelscited by 37
  11. 2023Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Textscited by 33
  12. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 31
38 more
  1. 2023Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Communitycited by 29
  2. 2024A Survey of Useful LLM Evaluationcited by 29
  3. 2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answeringcited by 28
  4. 2024UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMscited by 24
  5. 2024HyPA-RAG: A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applicationscited by 22
  6. 2023Memory Injections: Correcting Multi-Hop Reasoning Failures during Inference in Transformer-Based Language Modelscited by 21
  7. 2024MOPI-HFRS: A Multi-objective Personalized Health-aware Food Recommendation System with LLM-enhanced Interpretationcited by 18
  8. 2023From Text to Structure: Using Large Language Models to Support the Development of Legal Expert Systemscited by 17
  9. 2024CitaLaw: Enhancing LLM with Citations in Legal Domaincited by 16
  10. 2024Legal Judgment Reimagined: PredEx and the Rise of Intelligent AI Interpretation in Indian Courtscited by 15
  11. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 15
  12. 2024Diet-ODIN: A Novel Framework for Opioid Misuse Detection with Interpretable Dietary Patternscited by 14
  13. 2023Bringing order into the realm of Transformer-based language models for artificial intelligence and lawcited by 13
  14. 2023Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?cited by 12
  15. 2024Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understandingcited by 12
  16. 2023Legal Summarisation through LLMs: The PRODIGIT Projectcited by 10
  17. 2023BLT: Can Large Language Models Handle Basic Legal Text?cited by 10
  18. 2024Agents on the Bench: Large Language Model Based Multi Agent Framework for Trustworthy Digital Justicecited by 10
  19. 2025Large Language Models Meet Legal Artificial Intelligence: A Surveycited by 10
  20. 2024LLaMandement: Large Language Models for Summarization of French Legislative Proposalscited by 9
  21. 2025Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoningcited by 9
  22. 2025Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyondcited by 9
  23. 2024How Well Do LLMs Identify Cultural Unity in Diversity?cited by 8
  24. 2024Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attackscited by 8
  25. 2026Faithful Autoformalization via Roundtrip Verification and Repaircited by 8
  26. 2023LLMediator: GPT-4 Assisted Online Dispute Resolutioncited by 6
  27. 2023Black-Box Analysis: GPTs Across Time in Legal Textual Entailment Taskcited by 6
  28. 2024Exploring the Nexus of Large Language Models and Legal Systems: A Short Surveycited by 6
  29. 2024Evaluating Large Language Models for Public Health Classification and Extraction Taskscited by 6
  30. 2024Getting in the Door: Streamlining Intake in Civil Legal Services with Large Language Modelscited by 6
  31. 2024Information Re-Organization Improves Reasoning in Large Language Modelscited by 5
  32. 2024It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Humancited by 5
  33. 2023Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers?cited by 4
  34. 2023Retrieval-Augmented Chain-of-Thought in Semi-structured Domainscited by 4
  35. 2024When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challengescited by 4
  36. 2024Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluationscited by 4
  37. 2024Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Modelscited by 4
  38. 2024Software Engineering Methods For AI-Driven Deductive Legal Reasoningcited by 3

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Reframing Tax Law Entailment as Analogical Reasoning
  2. 2024Chain of Logic: Rule-Based Reasoning with Large Language Models
  3. 2023Generation of Explanations for Logic Reasoning
  4. 2025LawGPT: Knowledge-Guided Data Generation and Its Application to Legal LLM
  5. 2026GRACE: Grounded Adversarial Reasoning over Canadian Law
  6. 2026When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning
  7. 2025An Explicit Syllogistic Legal Reasoning Framework for Large Language Models
  8. 2025AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark