# Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation

arXiv: https://arxiv.org/abs/2603.06865 · HTML: https://mlsanity.com/paper/2603.06865/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 75 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Builds on (most cited first, 26 of 26)

- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators](https://mlsanity.com/paper/2403.16950/) (2024)
- [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023)
- [A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations](https://mlsanity.com/paper/2407.04069/) (2024)
- [Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation](https://mlsanity.com/paper/2208.11646/) (2022)
- [A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review](https://mlsanity.com/paper/2405.02559/) (2024)
- [Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing](https://mlsanity.com/paper/2104.10097/) (2021)
- [Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information](https://mlsanity.com/paper/2301.05036/) (2023)
- [Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis](https://mlsanity.com/paper/2308.16705/) (2023)
- [Analyzing Dataset Annotation Quality Management in the Wild](https://mlsanity.com/paper/2307.08153/) (2023)
- [You Are What You Annotate: Towards Better Models through Annotator Representations](https://mlsanity.com/paper/2305.14663/) (2023)
- [Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation](https://mlsanity.com/paper/2405.13769/) (2024)
- [Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability](https://mlsanity.com/paper/2106.07393/) (2021)
- [Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement](https://mlsanity.com/paper/2301.10684/) (2023)
- [Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks](https://mlsanity.com/paper/2212.09503/) (2022)
- [Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction](https://mlsanity.com/paper/1905.07791/) (2019)
- [Auditing Cross-Cultural Consistency of Human-Annotated Labels for Recommendation Systems](https://mlsanity.com/paper/2305.05917/) (2023)
- [Leveraging Annotator Disagreement for Text Classification](https://mlsanity.com/paper/2409.17577/) (2024)
- [k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations](https://mlsanity.com/paper/2203.12913/) (2022)
- [Compare without Despair: Reliable Preference Evaluation with Generation Separability](https://mlsanity.com/paper/2407.01878/) (2024)
- [DEXA: Supporting Non-Expert Annotators with Dynamic Examples from Experts](https://mlsanity.com/paper/2005.08367/) (2020)
- [Estimating Agreement by Chance for Sequence Annotation](https://mlsanity.com/paper/2407.11371/) (2024)
- [Towards Fair Pay and Equal Work: Imposing View Time Limits in Crowdsourced Image Classification](https://mlsanity.com/paper/2412.00260/) (2024)

## Built on by (most cited first, 13 of 13)

- [LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps](https://mlsanity.com/paper/2604.27345/) (2026)
- [The Compliance Gap: Why AI Systems Promise to Follow Process Instructions but Don't](https://mlsanity.com/paper/2605.01771/) (2026)
- [IntentGrasp: A Comprehensive Benchmark for Intent Understanding](https://mlsanity.com/paper/2605.06832/) (2026)
- [MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs](https://mlsanity.com/paper/2606.30026/) (2026)
- [Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation](https://mlsanity.com/paper/2608.00794/) (2026)
- [Automated IEP Generation from Traditional Chinese Parent-Teacher Interviews via Corpus-Grounded Feature Diffusion](https://mlsanity.com/paper/2606.09603/) (2026)
- [How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair](https://mlsanity.com/paper/2607.25873/) (2026)
- [Consensus Measures for Unstructured Biomedical Text Annotations](https://mlsanity.com/paper/2608.03529/) (2026)
- [GRACE:Gradient-guided Coreset Selection for LLM Unlearning](https://mlsanity.com/paper/2608.28361/) (2026)
- [Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis](https://mlsanity.com/paper/2608.30543/) (2026)
- [LLM Judge Validation Under Sparse Overlap: From Inference to Design](https://mlsanity.com/paper/2609.31857/) (2026)
- [VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation](https://mlsanity.com/paper/2609.35028/) (2026)
- [ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning](https://mlsanity.com/paper/2610.17183/) (2026)

## Abstract (arXiv)

Human annotation remains the foundation of reliable and interpretable data in Natural Language Processing (NLP). As annotation and evaluation tasks continue to expand, from categorical labelling to segmentation, subjective judgment, and continuous rating, measuring agreement between annotators has become increasingly more complex. This paper outlines how inter-annotator agreement (IAA) has been conceptualised and applied across NLP and related disciplines, describing the assumptions and limitations of common approaches. We organise agreement measures by task type and discuss how factors such as label imbalance and missing data influence reliability estimates. In addition, we highlight best practices for clear and transparent reporting, including the use of confidence intervals and the analysis of disagreement patterns. The paper aims to serve as a guide for selecting and interpreting agreement measures, promoting more consistent and reproducible human annotation and evaluation in NLP.