# AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference

arXiv: https://arxiv.org/abs/2302.00762 · HTML: https://mlsanity.com/paper/2302.00762/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 1 Feb 2023.

## Builds on (most cited first, 9 of 9)

- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [BERT for Coreference Resolution: Baselines and Analysis](https://mlsanity.com/paper/1908.09091/) (2019)
- [Deep Reinforcement Learning for Mention-Ranking Coreference Models](https://mlsanity.com/paper/1609.08667/) (2016)
- [A Crowdsourced Frame Disambiguation Corpus with Ambiguity](https://mlsanity.com/paper/1904.06101/) (2019)
- [The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generation](https://mlsanity.com/paper/2109.07848/) (2021)

## Built on by (most cited first, 12 of 12)

- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions](https://mlsanity.com/paper/2410.13788/) (2024)
- [Aligning Language Models to Explicitly Handle Ambiguity](https://mlsanity.com/paper/2404.11972/) (2024)
- [CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models](https://mlsanity.com/paper/2405.12063/) (2024)
- [AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries](https://mlsanity.com/paper/2406.19073/) (2024)
- [Zero and Few-shot Semantic Parsing with Ambiguous Inputs](https://mlsanity.com/paper/2306.00824/) (2023)
- [A Taxonomy of Ambiguity Types for NLP](https://mlsanity.com/paper/2403.14072/) (2024)
- [Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?](https://mlsanity.com/paper/2503.10838/) (2025)
- [Correct-Detect: Balancing Performance and Ambiguity Through the Lens of Coreference Resolution in LLMs](https://mlsanity.com/paper/2509.14456/) (2025)
- [Generative AI for Managerial Decision-Making under Ambiguity and Sycophancy](https://mlsanity.com/paper/2603.03970/) (2026)
- [Reasoning about Ambiguous Definite Descriptions](https://mlsanity.com/paper/2310.14657/) (2023)

## Abstract (arXiv)

Given a sentence "Abby told Brittney that she upset Courtney", one would struggle to understand who "she" refers to, and ask for clarification. However, if the word "upset" were replaced with "hugged", "she" unambiguously refers to Abby. We study if modern coreference resolution models are sensitive to such pronominal ambiguity. To this end, we construct AmbiCoref, a diagnostic corpus of minimal sentence pairs with ambiguous and unambiguous referents. Our examples generalize psycholinguistic studies of human perception of ambiguity around particular arrangements of verbs and their arguments. Analysis shows that (1) humans are less sure of referents in ambiguous AmbiCoref examples than unambiguous ones, and (2) most coreference models show little difference in output between ambiguous and unambiguous pairs. We release AmbiCoref as a diagnostic corpus for testing whether models treat ambiguity similarly to humans.