# Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks

arXiv: https://arxiv.org/abs/2212.09503 · HTML: https://mlsanity.com/paper/2212.09503/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 49 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 8 of 8)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation](https://mlsanity.com/paper/1707.05388/) (2017)
- [Data Excellence for AI: Why Should You Care](https://mlsanity.com/paper/2111.10391/) (2021)
- [Optimizing Open-Ended Crowdsourcing: The Next Frontier in Crowdsourced Data Management](https://mlsanity.com/paper/1610.05377/) (2016)

## Built on by (most cited first, 17 of 17)

- [Evaluation of African American Language Bias in Natural Language Generation](https://mlsanity.com/paper/2305.14291/) (2023)
- [Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research](https://mlsanity.com/paper/2412.04497/) (2024)
- [Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation](https://mlsanity.com/paper/2603.06865/) (2026)
- [Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages](https://mlsanity.com/paper/2506.00549/) (2025)
- [Evaluating AI systems under uncertain ground truth: a case study in dermatology](https://mlsanity.com/paper/2307.02191/) (2023)
- [Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection](https://mlsanity.com/paper/2308.12885/) (2023)
- [Clinical Annotations for Automatic Stuttering Severity Assessment](https://mlsanity.com/paper/2506.00644/) (2025)
- [CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reports](https://mlsanity.com/paper/2505.17265/) (2025)
- [NLP-based Decision Support System for Examination of Eligibility Criteria from Securities Prospectuses at the German Central Bank](https://mlsanity.com/paper/2302.04562/) (2023)
- [Bias in LLMs as Annotators: The Effect of Party Cues on Labelling Decision by Large Language Models](https://mlsanity.com/paper/2408.15895/) (2024)
- [MObyGaze: a film dataset of multimodal objectification densely annotated by experts](https://mlsanity.com/paper/2505.22084/) (2025)
- [All Required, In Order: Phase-Level Evaluation for AI-Human Dialogue in Healthcare and Beyond](https://mlsanity.com/paper/2601.08690/) (2026)
- [Enhancing Video Summarization with Context Awareness](https://mlsanity.com/paper/2404.04564/) (2024)
- [AI Competitions and Benchmarks: Dataset Development](https://mlsanity.com/paper/2404.09703/) (2024)
- [Construction and educational application of a linguistically grounded dependency treebank for Uyghur](https://mlsanity.com/paper/2507.21536/) (2025)
- [Consensus Measures for Unstructured Biomedical Text Annotations](https://mlsanity.com/paper/2608.03529/) (2026)
- [Quantifying Annotation Agreement in Object Detection with Optimal Transport](https://mlsanity.com/paper/2610.19912/) (2026)

## Abstract (arXiv)

When annotators label data, a key metric for quality assurance is inter-annotator agreement (IAA): the extent to which annotators agree on their labels. Though many IAA measures exist for simple categorical and ordinal labeling tasks, relatively little work has considered more complex labeling tasks, such as structured, multi-object, and free-text annotations. Krippendorff's alpha, best known for use with simpler labeling tasks, does have a distance-based formulation with broader applicability, but little work has studied its efficacy and consistency across complex annotation tasks. We investigate the design and evaluation of IAA measures for complex annotation tasks, with evaluation spanning seven diverse tasks: image bounding boxes, image keypoints, text sequence tagging, ranked lists, free text translations, numeric vectors, and syntax trees. We identify the difficulty of interpretability and the complexity of choosing a distance function as key obstacles in applying Krippendorff's alpha generally across these tasks. We propose two novel, more interpretable measures, showing they yield more consistent IAA measures across tasks and annotation distance functions.