# IdentifyMe: A Challenging Long-Context Mention Resolution Benchmark for LLMs

arXiv: https://arxiv.org/abs/2411.07466 · HTML: https://mlsanity.com/paper/2411.07466/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 19 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Builds on (most cited first, 5 of 5)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [An Annotated Dataset of Coreference in English Literature](https://mlsanity.com/paper/1912.01140/) (2019)
- [LingMess: Linguistically Informed Multi Expert Scorers for Coreference Resolution](https://mlsanity.com/paper/2205.12644/) (2022)
- [Are Large Language Models Robust Coreference Resolvers?](https://mlsanity.com/paper/2305.14489/) (2023)

## Built on by (most cited first, 4 of 4)

- [Automatic Generation of Inference Making Questions for Reading Comprehension Assessments](https://mlsanity.com/paper/2506.08260/) (2025)
- [ContractScrub: A benchmark for final review of legal contracts](https://mlsanity.com/paper/2608.20204/) (2026)
- [BiCon-Gate: Consistency-Gated De-colloquialisation for Dialogue Fact-Checking](https://mlsanity.com/paper/2604.14389/) (2026)
- [Human-Like Anaphor Resolution in Large Language Models](https://mlsanity.com/paper/2608.05630/) (2026)

## Abstract (arXiv)

Recent evaluations of LLMs on coreference resolution have revealed that traditional output formats and evaluation metrics do not fully capture the models' referential understanding. To address this, we introduce IdentifyMe, a new benchmark for mention resolution presented in a multiple-choice question (MCQ) format, commonly used for evaluating LLMs. IdentifyMe features long narratives and employs heuristics to exclude easily identifiable mentions, creating a more challenging task. The benchmark also consists of a curated mixture of different mention types and corresponding entities, allowing for a fine-grained analysis of model performance. We evaluate both closed- and open source LLMs on IdentifyMe and observe a significant performance gap (20-30%) between the state-of-the-art sub-10B open models vs. closed ones. We observe that pronominal mentions, which have limited surface information, are typically much harder for models to resolve than nominal mentions. Additionally, we find that LLMs often confuse entities when their mentions overlap in nested structures. The highest-scoring model, GPT-4o, achieves 81.9% accuracy, highlighting the strong referential capabilities of state-of-the-art LLMs while also indicating room for further improvement.