# Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts

arXiv: https://arxiv.org/abs/2305.04417 · HTML: https://mlsanity.com/paper/2305.04417/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 46 references that are in the map.
- First posted on arXiv 8 May 2023.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [Can GPT-3 Perform Statutory Reasoning?](https://mlsanity.com/paper/2302.06100/) (2023)
- [Legal Prompting: Teaching a Language Model to Think Like a Lawyer](https://mlsanity.com/paper/2212.01326/) (2022)
- [Identification of Rhetorical Roles of Sentences in Indian Legal Judgments](https://mlsanity.com/paper/1911.05405/) (2019)
- [Blind Judgement: Agent-Based Supreme Court Modelling With GPT](https://mlsanity.com/paper/2301.05327/) (2023)
- [MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding](https://mlsanity.com/paper/2301.00876/) (2023)
- [A Benchmark for Lease Contract Review](https://mlsanity.com/paper/2010.10386/) (2020)
- [How well do SOTA legal reasoning models support abductive reasoning?](https://mlsanity.com/paper/2304.06912/) (2023)
- [Computer-Assisted Creation of Boolean Search Rules for Text Classification in the Legal Domain](https://mlsanity.com/paper/2112.05807/) (2021)
- [Lex Rosetta: Transfer of Predictive Models Across Languages, Jurisdictions, and Legal Domains](https://mlsanity.com/paper/2112.07882/) (2021)
- [Sentence Embeddings and High-speed Similarity Search for Fast Computer Assisted Annotation of Legal Documents](https://mlsanity.com/paper/2112.11494/) (2021)
- [Performance in the Courtroom: Automated Processing and Visualization of Appeal Court Decisions in France](https://mlsanity.com/paper/2006.06251/) (2020)
- [Cross-Domain Generalization and Knowledge Transfer in Transformers Trained on Legal Data](https://mlsanity.com/paper/2112.07870/) (2021)

## Built on by (most cited first, 30 of 33)

- [Explaining Legal Concepts with Augmented Large Language Models (GPT-4)](https://mlsanity.com/paper/2306.09525/) (2023)
- [Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?](https://mlsanity.com/paper/2306.13906/) (2023)
- [Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies](https://mlsanity.com/paper/2310.18729/) (2023)
- [ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models](https://mlsanity.com/paper/2407.04693/) (2024)
- [Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?](https://mlsanity.com/paper/2310.14880/) (2023)
- [Large Language Models are legal but they are not: Making the case for a powerful LegalLLM](https://mlsanity.com/paper/2311.08890/) (2023)
- [CheX-GPT: Harnessing Large Language Models for Enhanced Chest X-ray Report Labeling](https://mlsanity.com/paper/2401.11505/) (2024)
- [From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews](https://mlsanity.com/paper/2312.01202/) (2023)
- [Harnessing GPT-3.5-turbo for Rhetorical Role Prediction in Legal Cases](https://mlsanity.com/paper/2310.17413/) (2023)
- [LLMediator: GPT-4 Assisted Online Dispute Resolution](https://mlsanity.com/paper/2307.16732/) (2023)
- [Getting in the Door: Streamlining Intake in Civil Legal Services with Large Language Models](https://mlsanity.com/paper/2410.03762/) (2024)
- [It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Human](https://mlsanity.com/paper/2407.06798/) (2024)
- [Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice](https://mlsanity.com/paper/2409.07713/) (2024)
- [Towards Trustworthy Legal AI through LLM Agents and Formal Reasoning](https://mlsanity.com/paper/2511.21033/) (2025)
- [Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers?](https://mlsanity.com/paper/2308.06032/) (2023)
- [Efficient Classification of Student Help Requests in Programming Courses Using Large Language Models](https://mlsanity.com/paper/2310.20105/) (2023)
- [GPT Assisted Annotation of Rhetorical and Linguistic Features for Interpretable Propaganda Technique Detection in News Text](https://mlsanity.com/paper/2407.11827/) (2024)
- [Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice](https://mlsanity.com/paper/2412.15260/) (2024)
- [Do Language Models Learn about Legal Entity Types during Pretraining?](https://mlsanity.com/paper/2310.13092/) (2023)
- [Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data](https://mlsanity.com/paper/2505.10260/) (2025)
- [Metadata Extraction Leveraging Large Language Models](https://mlsanity.com/paper/2510.19334/) (2025)
- [Physio: An LLM-Based Physiotherapy Advisor](https://mlsanity.com/paper/2401.01825/) (2024)
- [INACIA: Integrating Large Language Models in Brazilian Audit Courts: Opportunities and Challenges](https://mlsanity.com/paper/2401.05273/) (2024)
- [Investigating Low-Cost LLM Annotation for~Spoken Dialogue Understanding Datasets](https://mlsanity.com/paper/2406.13269/) (2024)
- [Can GPT-4 Help Detect Quit Vaping Intentions? An Exploration of Automatic Data Annotation Approach](https://mlsanity.com/paper/2407.00167/) (2024)
- [Impacts of Continued Legal Pre-Training and IFT on LLMs' Latent Representations of Human-Defined Legal Concepts](https://mlsanity.com/paper/2410.12001/) (2024)
- [Using Large Language Models for Legal Decision-Making in Austrian Value-Added Tax Law: A Comparative Study](https://mlsanity.com/paper/2507.08468/) (2025)
- [LaQual: An Automated Framework for LLM App Quality Evaluation](https://mlsanity.com/paper/2508.18636/) (2025)
- [Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law](https://mlsanity.com/paper/2605.16052/) (2026)
- [EURO-5K: When Does Domain Pretraining Matter? Benchmarking Transformers for EU Reporting Obligation Extraction](https://mlsanity.com/paper/2606.02971/) (2026)

## Abstract (arXiv)

We evaluated the capability of a state-of-the-art generative pre-trained transformer (GPT) model to perform semantic annotation of short text snippets (one to few sentences) coming from legal documents of various types. Discussions of potential uses (e.g., document drafting, summarization) of this emerging technology in legal domain have intensified, but to date there has not been a rigorous analysis of these large language models' (LLM) capacity in sentence-level semantic annotation of legal texts in zero-shot learning settings. Yet, this particular type of use could unlock many practical applications (e.g., in contract review) and research opportunities (e.g., in empirical legal studies). We fill the gap with this study. We examined if and how successfully the model can semantically annotate small batches of short text snippets (10-50) based exclusively on concise definitions of the semantic types. We found that the GPT model performs surprisingly well in zero-shot settings on diverse types of documents (F1=.73 on a task involving court opinions, .86 for contracts, and .54 for statutes and regulations). These findings can be leveraged by legal scholars and practicing lawyers alike to guide their decisions in integrating LLMs in wide range of workflows involving semantic annotation of legal texts.