mlsanityData to 7 Oct 2026

Sparse Probability of Agreement

arXiv 2208.06161 ↗First posted 12 Aug 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Measuring inter-annotator agreement is important for annotation tasks, but many metrics require a fully-annotated set of data, where all annotators annotate all samples. We define Sparse Probability of Agreement, SPA, which estimates the probability of agreement when not all annotator-item-pairs are available. We show that under certain conditions, SPA is an unbiased estimator, and we provide multiple weighing schemes for handling data with various degrees of annotation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 2 of its 11 references that are in the map. First posted on arXiv 12 Aug 2022.

Builds on

2 of 11 references.

References · 2

  1. 2018A Compact Embedding for Facial Expression Similaritycited by 41
  2. 2021MBIC -- A Media Bias Annotation Dataset Including Annotator Characteristicscited by 24

Built on by

Not cited yet.