# ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution

arXiv: https://arxiv.org/abs/2210.07188 · HTML: https://mlsanity.com/paper/2210.07188/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 66 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning](https://mlsanity.com/paper/1912.10389/) (2019)
- [An Annotated Dataset of Coreference in English Literature](https://mlsanity.com/paper/1912.01140/) (2019)
- [PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution](https://mlsanity.com/paper/1810.09807/) (2018)
- [Active Learning for Coreference Resolution using Discrete Annotation](https://mlsanity.com/paper/2004.13671/) (2020)
- [CoRefi: A Crowd Sourcing Suite for Coreference Annotation](https://mlsanity.com/paper/2010.02588/) (2020)
- [MOLEMAN: Mention-Only Linking of Entities with a Mention Annotation Network](https://mlsanity.com/paper/2106.07352/) (2021)
- [CD2CR: Co-reference Resolution Across Documents and Domains](https://mlsanity.com/paper/2101.12637/) (2021)
- [Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts](https://mlsanity.com/paper/2210.05581/) (2022)

## Built on by (most cited first, 2 of 2)

- [HateModerate: Testing Hate Speech Detectors against Content Moderation Policies](https://mlsanity.com/paper/2307.12418/) (2023)
- [CHAMP: Efficient Annotation and Consolidation of Cluster Hierarchies](https://mlsanity.com/paper/2311.11301/) (2023)

## Abstract (arXiv)

Large-scale, high-quality corpora are critical for advancing research in coreference resolution. However, existing datasets vary in their definition of coreferences and have been collected via complex and lengthy guidelines that are curated for linguistic experts. These concerns have sparked a growing interest among researchers to curate a unified set of guidelines suitable for annotators with various backgrounds. In this work, we develop a crowdsourcing-friendly coreference annotation methodology, ezCoref, consisting of an annotation tool and an interactive tutorial. We use ezCoref to re-annotate 240 passages from seven existing English coreference datasets (spanning fiction, news, and multiple other domains) while teaching annotators only cases that are treated similarly across these datasets. Surprisingly, we find that reasonable quality annotations were already achievable (>90% agreement between the crowd and expert annotations) even without extensive training. On carefully analyzing the remaining disagreements, we identify the presence of linguistic cases that our annotators unanimously agree upon but lack unified treatments (e.g., generic pronouns, appositives) in existing datasets. We propose the research community should revisit these phenomena when curating future unified annotation guidelines.