# AI on AI: Exploring the Utility of GPT as an Expert Annotator of AI Publications

arXiv: https://arxiv.org/abs/2403.09097 · HTML: https://mlsanity.com/paper/2403.09097/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 35 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 12 of 12)

- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts](https://mlsanity.com/paper/2205.01833/) (2022)
- [On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?](https://mlsanity.com/paper/2204.07931/) (2022)
- [On the Use of ArXiv as a Dataset](https://mlsanity.com/paper/1905.00075/) (2019)
- [When Will AI Exceed Human Performance? Evidence from AI Experts](https://mlsanity.com/paper/1705.08807/) (2017)
- [A Web-scale system for scientific knowledge exploration](https://mlsanity.com/paper/1805.12216/) (2018)
- [The AI Index 2021 Annual Report](https://mlsanity.com/paper/2103.06312/) (2021)
- [Long-Term Trends in the Public Perception of Artificial Intelligence](https://mlsanity.com/paper/1609.04904/) (2016)
- [Defining AI in Policy versus Practice](https://mlsanity.com/paper/1912.11095/) (2019)

## Built on by (most cited first, 4 of 4)

- [Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators](https://mlsanity.com/paper/2411.05854/) (2024)
- [Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers](https://mlsanity.com/paper/2408.03414/) (2024)
- [DeepDiveAI: Identifying AI Related Documents in Large Scale Literature Data](https://mlsanity.com/paper/2408.12871/) (2024)
- [MetaHarm: Harmful YouTube Video Dataset Annotated by Domain Experts, GPT-4-Turbo, and Crowdworkers](https://mlsanity.com/paper/2504.16304/) (2025)

## Abstract (arXiv)

Identifying scientific publications that are within a dynamic field of research often requires costly annotation by subject-matter experts. Resources like widely-accepted classification criteria or field taxonomies are unavailable for a domain like artificial intelligence (AI), which spans emerging topics and technologies. We address these challenges by inferring a functional definition of AI research from existing expert labels, and then evaluating state-of-the-art chatbot models on the task of expert data annotation. Using the arXiv publication database as ground-truth, we experiment with prompt engineering for GPT chatbot models to identify an alternative, automated expert annotation pipeline that assigns AI labels with 94% accuracy. For comparison, we fine-tune SPECTER, a transformer language model pre-trained on scientific publications, that achieves 96% accuracy (only 2% higher than GPT) on classifying AI publications. Our results indicate that with effective prompt engineering, chatbots can be used as reliable data annotators even where subject-area expertise is required. To evaluate the utility of chatbot-annotated datasets on downstream classification tasks, we train a new classifier on GPT-labeled data and compare its performance to the arXiv-trained model. The classifier trained on GPT-labeled data outperforms the arXiv-trained model by nine percentage points, achieving 82% accuracy.