# Estimating Agreement by Chance for Sequence Annotation

arXiv: https://arxiv.org/abs/2407.11371 · HTML: https://mlsanity.com/paper/2407.11371/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 5 of 5)

- [ExplainaBoard: An Explainable Leaderboard for NLP](https://mlsanity.com/paper/2104.06387/) (2021)
- [The Impact of Random Models on Clustering Similarity](https://mlsanity.com/paper/1701.06508/) (2017)
- [An Empirical Study into Annotator Agreement, Ground Truth Estimation, and Algorithm Evaluation](https://mlsanity.com/paper/1307.0426/) (2013)
- [What the F-measure doesn't measure: Features, Flaws, Fallacies and Fixes](https://mlsanity.com/paper/1503.06410/) (2015)
- [Understanding partition comparison indices based on counting object pairs](https://mlsanity.com/paper/1901.01777/) (2019)

## Built on by (most cited first, 2 of 2)

- [Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation](https://mlsanity.com/paper/2603.06865/) (2026)
- [Adjusted Similarity Measures and a Violation of Expectations](https://mlsanity.com/paper/2601.10641/) (2026)

## Abstract (arXiv)

In the field of natural language processing, correction of performance assessment for chance agreement plays a crucial role in evaluating the reliability of annotations. However, there is a notable dearth of research focusing on chance correction for assessing the reliability of sequence annotation tasks, despite their widespread prevalence in the field. To address this gap, this paper introduces a novel model for generating random annotations, which serves as the foundation for estimating chance agreement in sequence annotation tasks. Utilizing the proposed randomization model and a related comparison approach, we successfully derive the analytical form of the distribution, enabling the computation of the probable location of each annotated text segment and subsequent chance agreement estimation. Through a combination simulation and corpus-based evaluation, we successfully assess its applicability and validate its accuracy and efficacy.