mlsanityData to 7 Oct 2026

Legal Prompting: Teaching a Language Model to Think Like a Lawyer

arXiv 2212.01326 ↗First posted 2 Dec 2022cs.CL cs.AIcited by 60

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large language models that are capable of zero or few-shot prompting approaches have given rise to the new research area of prompt engineering. Recent advances showed that for example Chain-of-Thought (CoT) prompts can improve arithmetic or common sense tasks significantly. We explore how such approaches fare with legal reasoning tasks and take the COLIEE entailment task based on the Japanese Bar exam for testing zero-shot/few-shot and fine-tuning approaches. Our findings show that while CoT prompting and fine-tuning with explanations approaches show improvements, the best results are produced by prompts that are derived from specific legal reasoning techniques such as IRAC (Issue, Rule, Application, Conclusion). Based on our experiments we improve the 2021 best result from 0.7037 accuracy to 0.8148 accuracy and beat the 2022 best system of 0.6789 accuracy with an accuracy of 0.7431.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 23 references that are in the map. First posted on arXiv 2 Dec 2022.

Builds on

14 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Large Language Models are Zero-Shot Reasoners24 May

Other references · 13

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  9. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  10. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  11. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  12. 2021How Many Data Points is a Prompt Worth?cited by 197
1 more
  1. 2020LEGAL-BERT: The Muppets straight out of Law Schoolcited by 111

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 23; 2025: 9; 2026: 7.

60 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  3. 2023Challenges and Applications of Large Language Modelscited by 251
  4. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  5. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  6. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 133
  7. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  8. 2023Can GPT-3 Perform Statutory Reasoning?cited by 71
  9. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
  10. 2023A Short Survey of Viewing Large Language Models in Legal Aspectcited by 51
  11. 2023Explaining Legal Concepts with Augmented Large Language Models (GPT-4)cited by 39
  12. 2023Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Predictioncited by 37
38 more
  1. 2023Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Textscited by 32
  2. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  3. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 30
  4. 2024Chain of Logic: Rule-Based Reasoning with Large Language Modelscited by 18
  5. 2024A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industrycited by 17
  6. 2024CitaLaw: Enhancing LLM with Citations in Legal Domaincited by 16
  7. 2024Can Large Language Models Grasp Legal Theories? Enhance Legal Reasoning with Insights from Multi-Agent Collaborationcited by 14
  8. 2023The Importance of Prompt Tuning for Automated Neuron Explanationscited by 12
  9. 2023Legal Summarisation through LLMs: The PRODIGIT Projectcited by 10
  10. 2025Large Language Models Meet Legal Artificial Intelligence: A Surveycited by 9
  11. 2023Large Language Model Prompt Chaining for Long Legal Document Classificationcited by 7
  12. 2024Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunitiescited by 7
  13. 2023LLMediator: GPT-4 Assisted Online Dispute Resolutioncited by 6
  14. 2024Exploring the Nexus of Large Language Models and Legal Systems: A Short Surveycited by 6
  15. 2024Evaluating Large Language Models for Public Health Classification and Extraction Taskscited by 6
  16. 2024Logic Rules as Explanations for Legal Case Retrievalcited by 5
  17. 2024Enhancing Court View Generation with Knowledge Injection and Guidancecited by 5
  18. 2024Automating IRAC Analysis in Malaysian Contract Law using a Semi-Structured Knowledge Basecited by 5
  19. 2024Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justicecited by 5
  20. 2024Beyond Guilt: Legal Judgment Prediction with Trichotomous Reasoningcited by 5
  21. 2026LegalGraphRAG: Multi-Agent Graph Retrieval-Augmented Generation for Reliable Legal Reasoningcited by 5
  22. 2023Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers?cited by 4
  23. 2024When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challengescited by 4
  24. 2024Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documentscited by 4
  25. 2024HALO: Hallucination Analysis and Learning Optimization to Empower LLMs with Retrieval-Augmented Context for Guided Clinical Decision Makingcited by 3
  26. 2024Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justicecited by 3
  27. 2025LexChain: Modeling Legal Reasoning Chains for Chinese Tort Case Analysiscited by 3
  28. 2023Justifiable Artificial Intelligence: Engineering Large Language Models for Legal Applicationscited by 2
  29. 2024LAPIS: Language Model-Augmented Police Investigation Systemcited by 2
  30. 2025An Explicit Syllogistic Legal Reasoning Framework for Large Language Modelscited by 2
  31. 2025Reflexive Prompt Engineering: A Framework for Responsible Prompt Engineering and Interaction Designcited by 2
  32. 2023A Hierarchical Neural Framework for Classification and its Explanation in Large Unstructured Legal Documentscited by 1
  33. 2023A Search for Prompts: Generating Structured Answers from Contractscited by 1
  34. 2024Exploiting LLMs' Reasoning Capability to Infer Implicit Concepts in Legal Information Retrievalcited by 1
  35. 2024A Comprehensive Framework for Reliable Legal AI: Combining Specialized Expert Systems and Adaptive Refinementcited by 1
  36. 2025Aligning Language Models for Icelandic Legal Text Summarizationcited by 1
  37. 2026Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesiscited by 1
  38. 2023Making a Computational Attorneycited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve
  2. 2025Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting
  3. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
  4. 2025Effectiveness of Zero-shot-CoT in Japanese Prompts
  5. 2023Towards Better Chain-of-Thought Prompting Strategies: A Survey
  6. 2023Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning
  7. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
  8. 2023Large Language Models as Analogical Reasoners