# Who's Who: Large Language Models Meet Knowledge Conflicts in Practice

arXiv: https://arxiv.org/abs/2410.15737 · HTML: https://mlsanity.com/paper/2410.15737/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering](https://mlsanity.com/paper/2211.05655/) (2022)
- [Time Waits for No One! Analysis and Challenges of Temporal Misalignment](https://mlsanity.com/paper/2111.07408/) (2021)
- [Unfamiliar Finetuning Examples Control How Language Models Hallucinate](https://mlsanity.com/paper/2403.05612/) (2024)
- [Revisiting Challenges in Data-to-Text Generation with Fact Grounding](https://mlsanity.com/paper/2001.03830/) (2020)
- [Time Sensitive Knowledge Editing through Efficient Finetuning](https://mlsanity.com/paper/2406.04496/) (2024)

## Built on by (most cited first, 21 of 21)

- [M+: Extending MemoryLLM with Scalable Long-Term Memory](https://mlsanity.com/paper/2502.00592/) (2025)
- [STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?](https://mlsanity.com/paper/2605.06527/) (2026)
- [MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering](https://mlsanity.com/paper/2412.15540/) (2024)
- [What Makes a Good Natural Language Prompt?](https://mlsanity.com/paper/2506.06950/) (2025)
- [A Unified Definition of Hallucination: It's The World Model, Stupid!](https://mlsanity.com/paper/2512.21577/) (2025)
- [ConSens: Assessing context grounding in open-book question answering](https://mlsanity.com/paper/2505.00065/) (2025)
- [Format as a Prior: Quantifying and Analyzing Bias in LLMs for Heterogeneous Data](https://mlsanity.com/paper/2508.15793/) (2025)
- [Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence](https://mlsanity.com/paper/2601.11886/) (2026)
- [FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models](https://mlsanity.com/paper/2502.18573/) (2025)
- [Conflicts in Texts: Data, Implications and Challenges](https://mlsanity.com/paper/2504.19472/) (2025)
- [Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Models](https://mlsanity.com/paper/2504.08202/) (2025)
- [Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inference](https://mlsanity.com/paper/2606.20245/) (2026)
- [Information Discernment in Large Language Models](https://mlsanity.com/paper/2607.19355/) (2026)
- [ConceptCarve: Dynamic Realization of Evidence](https://mlsanity.com/paper/2504.07228/) (2025)
- [On Finding Inconsistencies in Documents](https://mlsanity.com/paper/2512.18601/) (2025)
- [ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence](https://mlsanity.com/paper/2606.26437/) (2026)
- [Hierarchical Graph Memory for LLM Agents with Path-level Localization and Rewrite](https://mlsanity.com/paper/2608.05095/) (2026)
- [Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge](https://mlsanity.com/paper/2608.28478/) (2026)
- [Identifying Temporal Features within Transcoders for Time Sensitive Factual Recall](https://mlsanity.com/paper/2609.33183/) (2026)
- [BELIEFRAG: Making Adaptive RAG State-Aware under Evolving Evidence](https://mlsanity.com/paper/2609.39139/) (2026)
- [Benchmarking and Training Reward Models for Cross-Source Evidence Grounding in Search Agents](https://mlsanity.com/paper/2610.17827/) (2026)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) methods are viable solutions for addressing the static memory limits of pre-trained language models. Nevertheless, encountering conflicting sources of information within the retrieval context is an inevitable practical challenge. In such situations, the language models are recommended to transparently inform users about the conflicts rather than autonomously deciding what to present based on their inherent biases. To analyze how current large language models (LLMs) align with our recommendation, we introduce WhoQA, a public benchmark dataset to examine model's behavior in knowledge conflict situations. We induce conflicts by asking about a common property among entities having the same name, resulting in questions with up to 8 distinctive answers. WhoQA evaluation set includes 5K questions across 13 Wikidata property types and 150K Wikipedia entities. Our experiments show that despite the simplicity of WhoQA questions, knowledge conflicts significantly degrades LLMs' performance in RAG settings.