mlsanityData to 7 Oct 2026

Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?

arXiv 2310.14880 ↗First posted 23 Oct 2023cs.CLcited by 12

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large Language Models (LLMs), such as ChatGPT, have drawn a lot of attentions recently in the legal domain due to its emergent ability to tackle a variety of legal tasks. However, it is still unknown if LLMs are able to analyze a legal case and perform reasoning in the same manner as lawyers. Therefore, we constructed a novel corpus consisting of scenarios pertain to Contract Acts Malaysia and Australian Social Act for Dependent Child. ChatGPT is applied to perform analysis on the corpus using the IRAC method, which is a framework widely used by legal professionals for organizing legal analysis. Each scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations. In addition, we conducted the first empirical assessment of ChatGPT for IRAC analysis in order to understand how well it aligns with the analysis of legal professionals. Our experimental results shed lights on possible future research directions to improve alignments between LLMs and legal experts in terms of legal reasoning.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 16 of its 24 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

16 of 24 references.

References · 16

  1. 2023Augmented Language Models: a Surveycited by 398
  2. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  3. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  4. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  5. 2023Auditing large language models: a three-layered approachcited by 135
  6. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  7. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  8. 2023Can GPT-3 Perform Statutory Reasoning?cited by 71
  9. 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
  10. 2022In conversation with Artificial Intelligence: aligning language models with human valuescited by 61
  11. 2023Natural Language Processing in the Legal Domaincited by 38
  12. 2023MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understandingcited by 32
4 more
  1. 2023Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Textscited by 32
  2. 2021Factoring Statutory Reasoning as Language Understanding Challengescited by 20
  3. 2021Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Paperscited by 18
  4. 2022LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 4; 2026: 2.

12 papers.

  1. 2024Prospects for inconsistency detection using large language models and sheavescited by 10
  2. 2025Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoningcited by 9
  3. 2024Automating IRAC Analysis in Malaysian Contract Law using a Semi-Structured Knowledge Basecited by 5
  4. 2024It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Humancited by 5
  5. 2025LRAGE: Legal Retrieval Augmented Generation Evaluation Toolcited by 5
  6. 2024Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Modelscited by 4
  7. 2025Taxation Perspectives from Large Language Models: A Case Study on Additional Tax Penaltiescited by 3
  8. 2026Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoningcited by 3
  9. 2024A Survey on the Real Power of ChatGPTcited by 0
  10. 2024Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?cited by 0
  11. 2025Aplicacc~ao de Large Language Models na An'alise e S'intese de Documentos Jur'idicos: Uma Revis~ao de Literaturacited by 0
  12. 2026LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers?
  2. 2025Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data
  3. 2025LawGPT: Knowledge-Guided Data Generation and Its Application to Legal LLM
  4. 2026Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases
  5. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions
  6. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence
  7. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  8. 2026Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task