mlsanityData to 7 Oct 2026

Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4

arXiv 2407.04130 ↗First posted 4 Jul 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper explores using GPT-3.5 and GPT-4 to automate the data annotation process with automatic prompting techniques. The main aim of this paper is to reuse human annotation guidelines along with some annotated data to design automatic prompts for LLMs, focusing on the semantic proximity annotation task. Automatic prompts are compared to customized prompts. We further implement the prompting strategies into an open-source text annotation tool, enabling easy online use via the OpenAI API. Our study reveals the crucial role of accurate prompt design and suggests that prompting GPT-4 with human-like instructions is not straightforwardly possible for the semantic proximity task. We show that small modifications to the human guidelines already improve the performance, suggesting possible ways for future research.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

24 of 34 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  6. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  7. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  8. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  9. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  10. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  11. 2022Is GPT-3 a Good Data Annotator?cited by 208
  12. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
12 more
  1. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 143
  2. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  3. 2020SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detectioncited by 105
  4. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97
  5. 2018Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Changecited by 35
  6. 2023Prompting Frameworks for Large Language Models: A Surveycited by 29
  7. 2021DWUG: A large Resource of Diachronic Word Usage Graphs in Four Languagescited by 25
  8. 2022Learning to Transfer Prompts for Text Generationcited by 22
  9. 2023Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligencecited by 13
  10. 2023SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domainscited by 12
  11. 2023Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?cited by 9
  12. 2024(Chat)GPT v BERT: Dawn of Justice for Semantic Change Detectioncited by 7

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 5.

6 papers.

  1. 2025From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labelingcited by 8
  2. 2024L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?cited by 5
  3. 2025CrowdAgent: Multi-Agent Managed Multi-Source Annotation Systemcited by 5
  4. 2025SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detectioncited by 2
  5. 2025Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotatorscited by 1
  6. 2025A Fully Automated Pipeline for Conversational Discourse Annotation: Tree Scheme Generation and Labeling with Large Language Modelscited by 1