# Towards Evaluation of Cross-document Coreference Resolution Models Using Datasets with Diverse Annotation Schemes

arXiv: https://arxiv.org/abs/2109.05250 · HTML: https://mlsanity.com/paper/2109.05250/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 39 references that are in the map.
- First posted on arXiv 11 Sep 2021.

## Builds on (most cited first, 4 of 4)

- [Revisiting Joint Modeling of Cross-document Entity and Event Coreference Resolution](https://mlsanity.com/paper/1906.01753/) (2019)
- [WEC: Deriving a Large-scale Cross-document Event Coreference dataset from Wikipedia](https://mlsanity.com/paper/2104.05022/) (2021)
- [Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora](https://mlsanity.com/paper/2011.12249/) (2020)
- [Realistic Evaluation Principles for Cross-document Coreference Resolution](https://mlsanity.com/paper/2106.04192/) (2021)

## Built on by (most cited first, 4 of 4)

- [A survey of diversity quantification in natural language processing: The why, what, where and how](https://mlsanity.com/paper/2507.20858/) (2025)
- [Code Book for the Annotation of Diverse Cross-Document Coreference of Entities in News Articles](https://mlsanity.com/paper/2310.12064/) (2023)
- [Diverse Word Choices, Same Reference: Annotating Lexically-Rich Cross-Document Coreference](https://mlsanity.com/paper/2602.17424/) (2026)
- [Piecing Together Cross-Document Coreference Resolution Datasets: Systematic Dataset Analysis and Unification](https://mlsanity.com/paper/2603.00621/) (2026)

## Abstract (arXiv)

Established cross-document coreference resolution (CDCR) datasets contain event-centric coreference chains of events and entities with identity relations. These datasets establish strict definitions of the coreference relations across related tests but typically ignore anaphora with more vague context-dependent loose coreference relations. In this paper, we qualitatively and quantitatively compare the annotation schemes of ECB+, a CDCR dataset with identity coreference relations, and NewsWCL50, a CDCR dataset with a mix of loose context-dependent and strict coreference relations. We propose a phrasing diversity metric (PD) that encounters for the diversity of full phrases unlike the previously proposed metrics and allows to evaluate lexical diversity of the CDCR datasets in a higher precision. The analysis shows that coreference chains of NewsWCL50 are more lexically diverse than those of ECB+ but annotating of NewsWCL50 leads to the lower inter-coder reliability. We discuss the different tasks that both CDCR datasets create for the CDCR models, i.e., lexical disambiguation and lexical diversity. Finally, to ensure generalizability of the CDCR models, we propose a direction for CDCR evaluation that combines CDCR datasets with multiple annotation schemes that focus of various properties of the coreference chains.