mlsanityData to 7 Oct 2026

Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation

arXiv 2603.06865 ↗First posted 6 Mar 2026cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Human annotation remains the foundation of reliable and interpretable data in Natural Language Processing (NLP). As annotation and evaluation tasks continue to expand, from categorical labelling to segmentation, subjective judgment, and continuous rating, measuring agreement between annotators has become increasingly more complex. This paper outlines how inter-annotator agreement (IAA) has been conceptualised and applied across NLP and related disciplines, describing the assumptions and limitations of common approaches. We organise agreement measures by task type and discuss how factors such as label imbalance and missing data influence reliability estimates. In addition, we highlight best practices for clear and transparent reporting, including the use of confidence intervals and the analysis of disagreement patterns. The paper aims to serve as a guide for selecting and interpreting agreement measures, promoting more consistent and reproducible human annotation and evaluation in NLP.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 26 of its 75 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

26 of 75 references.

References · 26

  1. 2024A Survey on LLM-as-a-Judgecited by 1,384
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  5. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  6. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  7. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  8. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  9. 2024A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Reviewcited by 65
  10. 2021Beyond Fair Pay: Ethical Implications of NLP Crowdsourcingcited by 52
  11. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  12. 2023Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysiscited by 44
14 more
  1. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  2. 2023Analyzing Dataset Annotation Quality Management in the Wildcited by 39
  3. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  4. 2021Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliabilitycited by 30
  5. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 22
  6. 2022Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Taskscited by 16
  7. 2019Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extractioncited by 9
  8. 2023Auditing Cross-Cultural Consistency of Human-Annotated Labels for Recommendation Systemscited by 9
  9. 2024Leveraging Annotator Disagreement for Text Classificationcited by 8
  10. 2022k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotationscited by 7
  11. 2024Compare without Despair: Reliable Preference Evaluation with Generation Separabilitycited by 4
  12. 2020DEXA: Supporting Non-Expert Annotators with Dynamic Examples from Expertscited by 3
  13. 2024Estimating Agreement by Chance for Sequence Annotationcited by 2
  14. 2024Towards Fair Pay and Equal Work: Imposing View Time Limits in Crowdsourced Image Classificationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gapscited by 4
  2. 2026IntentGrasp: A Comprehensive Benchmark for Intent Understandingcited by 1
  3. 2026MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMscited by 1
  4. 2026Automated IEP Generation from Traditional Chinese Parent-Teacher Interviews via Corpus-Grounded Feature Diffusioncited by 0
  5. 2026How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repaircited by 0
  6. 2026Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluationcited by 0
  7. 2026Consensus Measures for Unstructured Biomedical Text Annotationscited by 0
  8. 2026GRACE:Gradient-guided Coreset Selection for LLM Unlearningcited by 0
  9. 2026Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysiscited by 0
  10. 2026LLM Judge Validation Under Sparse Overlap: From Inference to Designcited by 0
  11. 2026VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformationcited by 0