# OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres

arXiv: https://arxiv.org/abs/2106.00933 · HTML: https://mlsanity.com/paper/2106.00933/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 28 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [Higher-order Coreference Resolution with Coarse-to-fine Inference](https://mlsanity.com/paper/1804.05392/) (2018)
- [BERT for Coreference Resolution: Baselines and Analysis](https://mlsanity.com/paper/1908.09091/) (2019)
- [Deep Reinforcement Learning for Mention-Ranking Coreference Models](https://mlsanity.com/paper/1609.08667/) (2016)
- [Improving Coreference Resolution by Learning Entity-Level Distributed Representations](https://mlsanity.com/paper/1606.01323/) (2016)
- [PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution](https://mlsanity.com/paper/1810.09807/) (2018)
- [Learning Global Features for Coreference Resolution](https://mlsanity.com/paper/1604.03035/) (2016)
- [Lexical Features in Coreference Resolution: To be Used With Caution](https://mlsanity.com/paper/1704.06779/) (2017)

## Built on by (most cited first, 18 of 18)

- [On Generalization in Coreference Resolution](https://mlsanity.com/paper/2109.09667/) (2021)
- [GENTLE: A Genre-Diverse Multilayer Challenge Set for English NLP and Linguistic Evaluation](https://mlsanity.com/paper/2306.01966/) (2023)
- [Why Can't Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversity](https://mlsanity.com/paper/2302.06488/) (2023)
- [Major Entity Identification: A Generalizable Alternative to Coreference Resolution](https://mlsanity.com/paper/2406.14654/) (2024)
- [Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End](https://mlsanity.com/paper/2212.10522/) (2022)
- [Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective](https://mlsanity.com/paper/2303.09092/) (2023)
- [GUMsley: Evaluating Entity Salience in Summarization for 12 English Genres](https://mlsanity.com/paper/2401.17974/) (2024)
- [GUM-SAGE: A Novel Dataset and Approach for Graded Entity Salience Prediction](https://mlsanity.com/paper/2504.10792/) (2025)
- [GDTB: Genre Diverse Data for English Shallow Discourse Parsing across Modalities, Text Types, and Domains](https://mlsanity.com/paper/2411.00491/) (2024)
- [A Controlled Reevaluation of Coreference Resolution Models](https://mlsanity.com/paper/2404.00727/) (2024)
- [Anatomy of OntoGUM--Adapting GUM to the OntoNotes Scheme to Evaluate Robustness of SOTA Coreference Algorithms](https://mlsanity.com/paper/2110.05727/) (2021)
- [What's Hard in English RST Parsing? Predictive Models for Error Analysis](https://mlsanity.com/paper/2309.04940/) (2023)
- [Unifying the Scope of Bridging Anaphora Types in English: Bridging Annotations in ARRAU and GUM](https://mlsanity.com/paper/2410.01170/) (2024)
- [Incorporating Singletons and Mention-based Features in Coreference Resolution via Multi-task Learning for Better Generalization](https://mlsanity.com/paper/2309.11582/) (2023)
- [Multimodal Coreference Resolution for Chinese Social Media Dialogues: Dataset and Benchmark Approach](https://mlsanity.com/paper/2504.14321/) (2025)
- [GUMBridge: a Corpus for Varieties of Bridging Anaphora](https://mlsanity.com/paper/2512.07134/) (2025)
- [SPLICE: A Singleton-Enhanced PipeLIne for Coreference REsolution](https://mlsanity.com/paper/2403.17245/) (2024)
- [Expect the Unexpected? Testing the Surprisal of Salient Entities](https://mlsanity.com/paper/2604.10724/) (2026)

## Abstract (arXiv)

SOTA coreference resolution produces increasingly impressive scores on the OntoNotes benchmark. However lack of comparable data following the same scheme for more genres makes it difficult to evaluate generalizability to open domain data. This paper provides a dataset and comprehensive evaluation showing that the latest neural LM based end-to-end systems degrade very substantially out of domain. We make an OntoNotes-like coreference dataset called OntoGUM publicly available, converted from GUM, an English corpus covering 12 genres, using deterministic rules, which we evaluate. Thanks to the rich syntactic and discourse annotations in GUM, we are able to create the largest human-annotated coreference corpus following the OntoNotes guidelines, and the first to be evaluated for consistency with the OntoNotes scheme. Out-of-domain evaluation across 12 genres shows nearly 15-20% degradation for both deterministic and deep learning systems, indicating a lack of generalizability or covert overfitting in existing coreference resolution models.