# Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective

arXiv: https://arxiv.org/abs/2303.09092 · HTML: https://mlsanity.com/paper/2303.09092/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 86 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 26 of 26)

- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning](https://mlsanity.com/paper/1908.05803/) (2019)
- [An Annotated Dataset of Coreference in English Literature](https://mlsanity.com/paper/1912.01140/) (2019)
- [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022)
- [Coreference Resolution through a seq2seq Transition-Based System](https://mlsanity.com/paper/2211.12142/) (2022)
- [Anaphora and Coreference Resolution: A Review](https://mlsanity.com/paper/1805.11824/) (2018)
- [The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks](https://mlsanity.com/paper/2210.10040/) (2022)
- [PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution](https://mlsanity.com/paper/1810.09807/) (2018)
- [LingMess: Linguistically Informed Multi Expert Scorers for Coreference Resolution](https://mlsanity.com/paper/2205.12644/) (2022)
- [On Generalization in Coreference Resolution](https://mlsanity.com/paper/2109.09667/) (2021)
- [Moving on from OntoNotes: Coreference Resolution Model Transfer](https://mlsanity.com/paper/2104.08457/) (2021)
- [OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres](https://mlsanity.com/paper/2106.00933/) (2021)
- [Findings of the Shared Task on Multilingual Coreference Resolution](https://mlsanity.com/paper/2209.07841/) (2022)
- [Lexical Features in Coreference Resolution: To be Used With Caution](https://mlsanity.com/paper/1704.06779/) (2017)
- [Improving Generalization in Coreference Resolution via Adversarial Training](https://mlsanity.com/paper/1908.04728/) (2019)
- [Using Automatically Extracted Minimum Spans to Disentangle Coreference Evaluation from Boundary Detection](https://mlsanity.com/paper/1906.06703/) (2019)
- [Solving Hard Coreference Problems](https://mlsanity.com/paper/1907.05524/) (2019)
- [A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and News](https://mlsanity.com/paper/2011.01615/) (2020)
- [The Gap on GAP: Tackling the Problem of Differing Data Distributions in Bias-Measuring Datasets](https://mlsanity.com/paper/2011.01837/) (2020)
- [Can we Fix the Scope for Coreference? Problems and Solutions for Benchmarks beyond OntoNotes](https://mlsanity.com/paper/2112.09742/) (2021)
- [Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts](https://mlsanity.com/paper/2210.05581/) (2022)
- [Multilingual Coreference Resolution in Multiparty Dialogue](https://mlsanity.com/paper/2208.01307/) (2022)

## Built on by (most cited first, 6 of 6)

- [CAW-coref: Conjunction-Aware Word-level Coreference Resolution](https://mlsanity.com/paper/2310.06165/) (2023)
- [Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor](https://mlsanity.com/paper/2506.14652/) (2025)
- [Semi-supervised multimodal coreference resolution in image narrations](https://mlsanity.com/paper/2310.13619/) (2023)
- [A Controlled Reevaluation of Coreference Resolution Models](https://mlsanity.com/paper/2404.00727/) (2024)
- [Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems](https://mlsanity.com/paper/2506.04482/) (2025)
- [Interpretable Coreference Resolution Evaluation Using Explicit Semantics](https://mlsanity.com/paper/2605.10627/) (2026)

## Abstract (arXiv)

It is increasingly common to evaluate the same coreference resolution (CR) model on multiple datasets. Do these multi-dataset evaluations allow us to draw meaningful conclusions about model generalization? Or, do they rather reflect the idiosyncrasies of a particular experimental setup (e.g., the specific datasets used)? To study this, we view evaluation through the lens of measurement modeling, a framework commonly used in the social sciences for analyzing the validity of measurements. By taking this perspective, we show how multi-dataset evaluations risk conflating different factors concerning what, precisely, is being measured. This in turn makes it difficult to draw more generalizable conclusions from these evaluations. For instance, we show that across seven datasets, measurements intended to reflect CR model generalization are often correlated with differences in both how coreference is defined and how it is operationalized; this limits our ability to draw conclusions regarding the ability of CR models to generalize across any singular dimension. We believe the measurement modeling framework provides the needed vocabulary for discussing challenges surrounding what is actually being measured by CR evaluations.