mlsanityData to 14 Oct 2026

Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators

arXiv 2503.06778 ↗First posted 9 Mar 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Event annotation is important for identifying market changes, monitoring breaking news, and understanding sociological trends. Although expert annotators set the gold standards, human coding is expensive and inefficient. Unlike information extraction experiments that focus on single contexts, we evaluate a holistic workflow that removes irrelevant documents, merges documents about the same event, and annotates the events. Although LLM-based automated annotations are better than traditional TF-IDF-based methods or Event Set Curation, they are still not reliable annotators compared to human experts. However, adding LLMs to assist experts for Event Set Curation can reduce the time and mental effort required for Variable Annotation. When using LLMs to extract event variables to assist expert annotators, they agree more with the extracted variables than fully automated LLMs for annotation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 88 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

33 of 88 references.

References · 33

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  4. 2015Teaching Machines to Read and Comprehendcited by 1,049
  5. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  6. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  7. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  8. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  9. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  10. 2021Document-Level Event Argument Extraction by Conditional Generationcited by 155
  11. 2018Learning Sequence Encoders for Temporal Knowledge Graph Completioncited by 145
  12. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
21 more
  1. 2018Text Segmentation as a Supervised Learning Taskcited by 76
  2. 2019Revisiting Joint Modeling of Cross-document Entity and Event Coreference Resolutioncited by 43
  3. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  4. 2020Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentationcited by 41
  5. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  6. 2021Cross-document Coreference Resolution over Predicted Mentionscited by 29
  7. 2022SODAPOP: Open-Ended Discovery of Social Biases in Social Commonsense Reasoning Modelscited by 22
  8. 2020Generalizing Cross-Document Event Coreference Resolution Across Multiple Corporacited by 18
  9. 2020Streamlining Cross-Document Coreference Resolution: Evaluation and Modelingcited by 16
  10. 2021Realistic Evaluation Principles for Cross-document Coreference Resolutioncited by 15
  11. 2020CoRefi: A Crowd Sourcing Suite for Coreference Annotationcited by 12
  12. 2022Cross-document Event Coreference Search: Task, Dataset and Modelingcited by 12
  13. 2024ULTRA: Unleash LLMs' Potential for Event Argument Extraction through Hierarchical Modeling and Pair-wise Self-Refinementcited by 12
  14. 2024Beyond Single-Event Extraction: Towards Efficient Document-Level Multi-Event Argument Extractioncited by 12
  15. 2020Structural Text Segmentation of Legal Documentscited by 11
  16. 2024Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysiscited by 10
  17. 2024Explicit, Implicit, and Scattered: Revisiting Event Extraction to Capture Complex Argumentscited by 9
  18. 2024Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4cited by 6
  19. 2024Generating Harder Cross-document Event Coreference Resolution Datasets using Metaphoric Paraphrasingcited by 6
  20. 2025Summaries as Centroids for Interpretable and Scalable Text Clusteringcited by 4
  21. 2024Harvesting Events from Multiple Sources: Towards a Cross-Document Event Extraction Paradigmcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Are LLMs Ready to Replace Bangla Annotators?cited by 0
  2. 2026Towards Cost-Efficient Video Dataset Construction with Large Language Modelscited by 0