# Large corpora and large language models: a replicable method for automating grammatical annotation

arXiv: https://arxiv.org/abs/2411.11260 · HTML: https://mlsanity.com/paper/2411.11260/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 60 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Can Large Language Models Reason and Plan?](https://mlsanity.com/paper/2403.04121/) (2024)
- [Mapping Languages: The Corpus of Global Language Use](https://mlsanity.com/paper/2004.00798/) (2020)
- [Leveraging ChatGPT As Text Annotation Tool For Sentiment Analysis](https://mlsanity.com/paper/2306.17177/) (2023)
- [The Sociolinguistic Foundations of Language Modeling](https://mlsanity.com/paper/2407.09241/) (2024)
- [Can Large Language Models Reason? A Characterization via 3-SAT](https://mlsanity.com/paper/2408.07215/) (2024)
- [Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis: The case of apology](https://mlsanity.com/paper/2305.08339/) (2023)
- [Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model](https://mlsanity.com/paper/2310.15113/) (2023)
- [Exposure and Emergence in Usage-Based Grammar: Computational Experiments in 35 Languages](https://mlsanity.com/paper/2211.14160/) (2022)

## Built on by (most cited first, 5 of 5)

- [Linguistics and Human Brain: A Perspective of Computational Neuroscience](https://mlsanity.com/paper/2602.08275/) (2026)
- [Metaphor identification using large language models: A comparison of RAG, prompt engineering, and fine-tuning](https://mlsanity.com/paper/2509.24866/) (2025)
- [TACOMORE: Exploring a replicable prompting protocol for LLM-assisted corpus analysis](https://mlsanity.com/paper/2412.10139/) (2024)
- [LLMs Learn Constructions That Humans Do Not Know](https://mlsanity.com/paper/2508.16837/) (2025)
- [LLMs for automatic annotation of Mandarin narrative transcripts](https://mlsanity.com/paper/2605.17205/) (2026)

## Abstract (arXiv)

Much linguistic research relies on annotated datasets of features extracted from text corpora, but the rapid quantitative growth of these corpora has created practical difficulties for linguists to manually annotate large data samples. In this paper, we present a replicable, supervised method that leverages large language models for assisting the linguist in grammatical annotation through prompt engineering, training, and evaluation. We introduce a methodological pipeline applied to the case study of formal variation in the English evaluative verb construction 'consider X (as) (to be) Y', based on the large language model Claude 3.5 Sonnet and corpus data from Davies' NOW and EnTenTen21 (SketchEngine). Overall, we reach a model accuracy of over 90% on our held-out test samples with only a small amount of training data, validating the method for the annotation of very large quantities of tokens of the construction in the future. We discuss the generalisability of our results for a wider range of case studies of grammatical constructions and grammatical variation and change, underlining the value of AI copilots as tools for future linguistic research, notwithstanding some important caveats.