# AmbigDocs: Reasoning across Documents on Different Entities under the Same Name

arXiv: https://arxiv.org/abs/2404.12447 · HTML: https://mlsanity.com/paper/2404.12447/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 48 references that are in the map.
- First posted on arXiv 18 Apr 2024.

## Builds on (most cited first, 30 of 40)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering](https://mlsanity.com/paper/2307.16877/) (2023)
- [Fine-grained Hallucination Detection and Editing for Language Models](https://mlsanity.com/paper/2401.06855/) (2024)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence](https://mlsanity.com/paper/2210.13701/) (2022)
- [End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering](https://mlsanity.com/paper/2106.05346/) (2021)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [What Evidence Do Language Models Find Convincing?](https://mlsanity.com/paper/2402.11782/) (2024)
- [Reliable, Adaptable, and Attributable Language Models with Retrieval](https://mlsanity.com/paper/2403.03187/) (2024)
- [CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models](https://mlsanity.com/paper/2212.07769/) (2022)
- [Selectively Answering Ambiguous Questions](https://mlsanity.com/paper/2305.14613/) (2023)
- [Can NLI Models Verify QA Systems' Predictions?](https://mlsanity.com/paper/2104.08731/) (2021)
- [Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2310.14696/) (2023)

## Built on by (most cited first, 21 of 21)

- [Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions](https://mlsanity.com/paper/2410.13788/) (2024)
- [SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models](https://mlsanity.com/paper/2506.01062/) (2025)
- [Retrieval-Augmented Generation with Conflicting Evidence](https://mlsanity.com/paper/2504.13079/) (2025)
- [Molecular Facts: Desiderata for Decontextualization in LLM Fact Verification](https://mlsanity.com/paper/2406.20079/) (2024)
- [TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.00879/) (2025)
- [DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs](https://mlsanity.com/paper/2506.08500/) (2025)
- [DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation](https://mlsanity.com/paper/2412.13175/) (2024)
- [CodeUpdateArena: Benchmarking Knowledge Editing on API Updates](https://mlsanity.com/paper/2407.06249/) (2024)
- [AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web](https://mlsanity.com/paper/2505.17978/) (2025)
- [Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs](https://mlsanity.com/paper/2505.17762/) (2025)
- [Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents](https://mlsanity.com/paper/2605.09698/) (2026)
- [Can Large Language Models Really Recognize Your Name?](https://mlsanity.com/paper/2505.14549/) (2025)
- [MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance](https://mlsanity.com/paper/2506.14927/) (2025)
- [DBRouting: Routing End User Queries to Databases for Answerability](https://mlsanity.com/paper/2501.16220/) (2025)
- [Agentic Verification for Ambiguous Query Disambiguation](https://mlsanity.com/paper/2502.10352/) (2025)
- [Inter-Passage Verification for Multi-evidence Multi-answer QA](https://mlsanity.com/paper/2506.00425/) (2025)
- [MARCH: Evaluating the Intersection of Ambiguity Interpretation and Multi-hop Inference](https://mlsanity.com/paper/2509.22750/) (2025)
- [One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies](https://mlsanity.com/paper/2607.21143/) (2026)
- [ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence](https://mlsanity.com/paper/2606.26437/) (2026)
- [Dual-Confidence Contrastive Decoding for Retrieval-Augmented Generation](https://mlsanity.com/paper/2607.00570/) (2026)
- [When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain](https://mlsanity.com/paper/2609.24290/) (2026)

## Abstract (arXiv)

Different entities with the same name can be difficult to distinguish. Handling confusing entity mentions is a crucial skill for language models (LMs). For example, given the question "Where was Michael Jordan educated?" and a set of documents discussing different people named Michael Jordan, can LMs distinguish entity mentions to generate a cohesive answer to the question? To test this ability, we introduce a new benchmark, AmbigDocs. By leveraging Wikipedia's disambiguation pages, we identify a set of documents, belonging to different entities who share an ambiguous name. From these documents, we generate questions containing an ambiguous name and their corresponding sets of answers. Our analysis reveals that current state-of-the-art models often yield ambiguous answers or incorrectly merge information belonging to different entities. We establish an ontology categorizing four types of incomplete answers and automatic evaluation metrics to identify such categories. We lay the foundation for future work on reasoning across multiple documents with ambiguous entities.