mlsanityData to 7 Oct 2026

LLMs as Span Annotators: A Comparative Study of LLMs and Humans

arXiv 2504.08697 ↗First posted 11 Apr 2025cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Span annotation - annotating specific text features at the span level - can be used to evaluate texts where single-score metrics fail to provide actionable feedback. Until recently, span annotation was done by human annotators or fine-tuned models. In this paper, we study whether large language models (LLMs) can serve as an alternative to human annotators. We compare the abilities of LLMs to skilled human annotators on three span annotation tasks: evaluating data-to-text generation, identifying translation errors, and detecting propaganda techniques. We show that overall, LLMs have only moderate inter-annotator agreement (IAA) with human annotators. However, we demonstrate that LLMs make errors at a similar rate as skilled crowdworkers. LLMs also produce annotations at a fraction of the cost per output annotation. We release the dataset of over 40k model and human span annotations for further research.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 39 of its 72 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

39 of 72 references.

References · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2024A Survey on LLM-as-a-Judgecited by 1,384
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  9. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  10. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  11. 2020Evaluation of Text Generation: A Surveycited by 275
  12. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
27 more
  1. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  2. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  3. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  4. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  5. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  6. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  7. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  8. 2019Fine-Grained Analysis of Propaganda in News Articlescited by 130
  9. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  10. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  11. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  12. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 91
  13. 2020SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articlescited by 79
  14. 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
  15. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  16. 2020A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systemscited by 34
  17. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  18. 2024AI-Assisted Human Evaluation of Machine Translationcited by 20
  19. 2024Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writerscited by 19
  20. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
  21. 2023Incorporating Worker Perspectives into MTurk Annotation Practices for NLPcited by 17
  22. 2024Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generationcited by 16
  23. 2020Best Practices for Managing Data Annotation Projectscited by 12
  24. 2022Innovations in Neural Data-to-text Generation: A Surveycited by 10
  25. 2022Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarizationcited by 10
  26. 2025Fine-grained Fallacy Detection with Human Label Variationcited by 6
  27. 2025OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025FLAWS: A Benchmark for Error Identification and Localization in Scientific Paperscited by 12
  2. 2026Strategies for Span Labeling with Large Language Modelscited by 2
  3. 2025Towards Temporal Knowledge-Base Creation for Fine-Grained Opinion Analysis with Language Modelscited by 1
  4. 2025Real-World Summarization: When Evaluation Reaches Its Limitscited by 0
  5. 2026Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysiscited by 0
  6. 2026JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errorscited by 0
  7. 2026Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challengedcited by 0