# Sparse Probability of Agreement

arXiv: https://arxiv.org/abs/2208.06161 · HTML: https://mlsanity.com/paper/2208.06161/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 11 references that are in the map.
- First posted on arXiv 12 Aug 2022.

## Builds on (most cited first, 2 of 2)

- [A Compact Embedding for Facial Expression Similarity](https://mlsanity.com/paper/1811.11283/) (2018)
- [MBIC -- A Media Bias Annotation Dataset Including Annotator Characteristics](https://mlsanity.com/paper/2105.11910/) (2021)

## Abstract (arXiv)

Measuring inter-annotator agreement is important for annotation tasks, but many metrics require a fully-annotated set of data, where all annotators annotate all samples. We define Sparse Probability of Agreement, SPA, which estimates the probability of agreement when not all annotator-item-pairs are available. We show that under certain conditions, SPA is an unbiased estimator, and we provide multiple weighing schemes for handling data with various degrees of annotation.