# Realistic Evaluation Principles for Cross-document Coreference Resolution

arXiv: https://arxiv.org/abs/2106.04192 · HTML: https://mlsanity.com/paper/2106.04192/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 26 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 10 of 10)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [BERT for Coreference Resolution: Baselines and Analysis](https://mlsanity.com/paper/1908.09091/) (2019)
- [Revisiting Joint Modeling of Cross-document Entity and Event Coreference Resolution](https://mlsanity.com/paper/1906.01753/) (2019)
- [PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution](https://mlsanity.com/paper/1810.09807/) (2018)
- [Cross-document Coreference Resolution over Predicted Mentions](https://mlsanity.com/paper/2106.01210/) (2021)
- [Resolving Event Coreference with Supervised Representation Learning and Clustering-Oriented Regularization](https://mlsanity.com/paper/1805.10985/) (2018)
- [Event Coreference Resolution by Iteratively Unfolding Inter-dependencies among Events](https://mlsanity.com/paper/1707.07344/) (2017)
- [Paraphrasing vs Coreferring: Two Sides of the Same Coin](https://mlsanity.com/paper/2004.14979/) (2020)
- [A Hierarchical Distance-dependent Bayesian Model for Event Coreference Resolution](https://mlsanity.com/paper/1504.05929/) (2015)

## Built on by (most cited first, 15 of 15)

- [What happens before and after: Multi-Event Commonsense in Event Coreference Resolution](https://mlsanity.com/paper/2302.09715/) (2023)
- [Contrastive Representation Learning for Cross-Document Coreference Resolution of Events and Entities](https://mlsanity.com/paper/2205.11438/) (2022)
- [MDCure: A Scalable Pipeline for Multi-Document Instruction-Following](https://mlsanity.com/paper/2410.23463/) (2024)
- [Text-based NP Enrichment](https://mlsanity.com/paper/2109.12085/) (2021)
- [SEAM: A Stochastic Benchmark for Multi-Document Tasks](https://mlsanity.com/paper/2406.16086/) (2024)
- [EDIN: An End-to-end Benchmark and Pipeline for Unknown Entity Discovery and Indexing](https://mlsanity.com/paper/2205.12570/) (2022)
- [Towards Evaluation of Cross-document Coreference Resolution Models Using Datasets with Diverse Annotation Schemes](https://mlsanity.com/paper/2109.05250/) (2021)
- [Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models](https://mlsanity.com/paper/2406.02148/) (2024)
- [Cross-document Event Identity via Dense Annotation](https://mlsanity.com/paper/2109.06417/) (2021)
- [Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators](https://mlsanity.com/paper/2503.06778/) (2025)
- [Dense Paraphrasing for Textual Enrichment](https://mlsanity.com/paper/2210.11563/) (2022)
- [Link Prediction for Event Logs in the Process Industry](https://mlsanity.com/paper/2508.09096/) (2025)
- [Filling in the Gaps: Efficient Event Coreference Resolution using Graph Autoencoder Networks](https://mlsanity.com/paper/2310.11965/) (2023)
- [Piecing Together Cross-Document Coreference Resolution Datasets: Systematic Dataset Analysis and Unification](https://mlsanity.com/paper/2603.00621/) (2026)
- [Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text](https://mlsanity.com/paper/2604.17108/) (2026)

## Abstract (arXiv)

We point out that common evaluation practices for cross-document coreference resolution have been unrealistically permissive in their assumed settings, yielding inflated results. We propose addressing this issue via two evaluation methodology principles. First, as in other tasks, models should be evaluated on predicted mentions rather than on gold mentions. Doing this raises a subtle issue regarding singleton coreference clusters, which we address by decoupling the evaluation of mention detection from that of coreference linking. Second, we argue that models should not exploit the synthetic topic structure of the standard ECB+ dataset, forcing models to confront the lexical ambiguity challenge, as intended by the dataset creators. We demonstrate empirically the drastic impact of our more realistic evaluation principles on a competitive model, yielding a score which is 33 F1 lower compared to evaluating by prior lenient practices.