# Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis: The case of apology

arXiv: https://arxiv.org/abs/2305.08339 · HTML: https://mlsanity.com/paper/2305.08339/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 51 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 7 of 7)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023)

## Built on by (most cited first, 10 of 10)

- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [Large corpora and large language models: a replicable method for automating grammatical annotation](https://mlsanity.com/paper/2411.11260/) (2024)
- [Linguistics and Human Brain: A Perspective of Computational Neuroscience](https://mlsanity.com/paper/2602.08275/) (2026)
- [Metaphor identification using large language models: A comparison of RAG, prompt engineering, and fine-tuning](https://mlsanity.com/paper/2509.24866/) (2025)
- [Can GPT-4 learn to analyse moves in research article abstracts?](https://mlsanity.com/paper/2407.15612/) (2024)
- [TACOMORE: Exploring a replicable prompting protocol for LLM-assisted corpus analysis](https://mlsanity.com/paper/2412.10139/) (2024)
- [Pragmatics beyond humans: meaning, communication, and LLMs](https://mlsanity.com/paper/2508.06167/) (2025)
- [Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts](https://mlsanity.com/paper/2604.03127/) (2026)
- [LLMs for automatic annotation of Mandarin narrative transcripts](https://mlsanity.com/paper/2605.17205/) (2026)
- [Challenges in annotations by humans and LLMs: A case study of evaluative language](https://mlsanity.com/paper/2607.28119/) (2026)

## Abstract (arXiv)

Certain forms of linguistic annotation, like part of speech and semantic tagging, can be automated with high accuracy. However, manual annotation is still necessary for complex pragmatic and discursive features that lack a direct mapping to lexical forms. This manual process is time-consuming and error-prone, limiting the scalability of function-to-form approaches in corpus linguistics. To address this, our study explores the possibility of using large language models (LLMs) to automate pragma-discursive corpus annotation. We compare GPT-3.5 (the model behind the free-to-use version of ChatGPT), GPT-4 (the model underpinning the precise mode of Bing chatbot), and a human coder in annotating apology components in English based on the local grammar framework. We find that GPT-4 outperformed GPT-3.5, with accuracy approaching that of a human coder. These results suggest that LLMs can be successfully deployed to aid pragma-discursive corpus annotation, making the process more efficient, scalable and accessible.