# DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Models

arXiv: https://arxiv.org/abs/2407.17023 · HTML: https://mlsanity.com/paper/2407.17023/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [On the Risk of Misinformation Pollution with Large Language Models](https://mlsanity.com/paper/2305.13661/) (2023)
- [TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models](https://mlsanity.com/paper/2204.14211/) (2022)
- [The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation](https://mlsanity.com/paper/2312.09085/) (2023)
- [DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering](https://mlsanity.com/paper/2211.05655/) (2022)
- [How Do Large Language Models Acquire Factual Knowledge During Pretraining?](https://mlsanity.com/paper/2406.11813/) (2024)
- [What Evidence Do Language Models Find Convincing?](https://mlsanity.com/paper/2402.11782/) (2024)
- [When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation](https://mlsanity.com/paper/2402.11457/) (2024)
- [SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models](https://mlsanity.com/paper/2403.02509/) (2024)
- [Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views](https://mlsanity.com/paper/2302.12297/) (2023)
- [Mitigating Temporal Misalignment by Discarding Outdated Facts](https://mlsanity.com/paper/2305.14824/) (2023)
- [MuLan: A Study of Fact Mutability in Language Models](https://mlsanity.com/paper/2404.03036/) (2024)
- [Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods](https://mlsanity.com/paper/2404.18655/) (2024)
- [Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models](https://mlsanity.com/paper/2402.15162/) (2024)

## Built on by (most cited first, 30 of 34)

- [Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checking](https://mlsanity.com/paper/2502.09083/) (2025)
- [To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts](https://mlsanity.com/paper/2410.14675/) (2024)
- [Claim Verification in the Age of Large Language Models: A Survey](https://mlsanity.com/paper/2408.14317/) (2024)
- [EvoWiki: Evaluating LLMs on Evolving Knowledge](https://mlsanity.com/paper/2412.13582/) (2024)
- [Whose Facts Win? LLM Source Preferences under Knowledge Conflicts](https://mlsanity.com/paper/2601.03746/) (2026)
- [A Reality Check on Context Utilisation for Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.17031/) (2024)
- [Stochastic Chameleons: Irrelevant Context Hallucinations Reveal Class-Based (Mis)Generalization in LLMs](https://mlsanity.com/paper/2505.22630/) (2025)
- [MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.21544/) (2025)
- [On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2502.14759/) (2025)
- [Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Models](https://mlsanity.com/paper/2508.19720/) (2025)
- [Where Knowledge Collides: A Mechanistic Study of Intra-Memory Knowledge Conflict in Language Models](https://mlsanity.com/paper/2601.09445/) (2026)
- [CUB: Benchmarking Context Utilisation Techniques for Language Models](https://mlsanity.com/paper/2505.16518/) (2025)
- [Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large Language Models](https://mlsanity.com/paper/2509.23233/) (2025)
- [KScope: A Framework for Characterizing the Knowledge Status of Language Models](https://mlsanity.com/paper/2506.07458/) (2025)
- [A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models](https://mlsanity.com/paper/2409.15228/) (2024)
- [FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models](https://mlsanity.com/paper/2502.18573/) (2025)
- [When Facts Change: Probing LLMs on Evolving Knowledge with evolveQA](https://mlsanity.com/paper/2510.19172/) (2025)
- [Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.14546/) (2026)
- [Temporal Fact Conflicts in LLMs: Reproducibility Insights from Unifying DYNAMICQA and MULAN](https://mlsanity.com/paper/2603.15892/) (2026)
- [Three Regimes of Context-Parametric Conflict: A Predictive Framework and Empirical Validation](https://mlsanity.com/paper/2605.11574/) (2026)
- [Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Models](https://mlsanity.com/paper/2606.20959/) (2026)
- [Evidence-Grounded Verified Agentic Reasoning: A Path Toward Eliminating LLM Hallucination in Empirical Inference via Tool-Attested Kernel Proofs](https://mlsanity.com/paper/2607.12650/) (2026)
- [Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace Disentanglement](https://mlsanity.com/paper/2511.01706/) (2025)
- [How Context Shapes Truth: Geometric Transformations of Statement-level Truth Representations in LLMs](https://mlsanity.com/paper/2601.06599/) (2026)
- [Understanding the Interplay between LLMs' Utilisation of Parametric and Contextual Knowledge: A keynote at ECIR 2025](https://mlsanity.com/paper/2603.09654/) (2026)
- [Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answering](https://mlsanity.com/paper/2604.07985/) (2026)
- [Diagnosing LLM Arbitration Behavior over Pre-evidence Epistemic States in RAG-based Fact-Checking](https://mlsanity.com/paper/2606.01120/) (2026)
- [When Personal Memory Has No Single Answer: Evaluating LLM Agents under Irreducible Conflict](https://mlsanity.com/paper/2608.13921/) (2026)
- [Prompt-Based Abstention Fails Under Misleading Context: A Controlled Study of Small Frozen RAG Models](https://mlsanity.com/paper/2608.22228/) (2026)
- [Towards Evolving Context Parameterization for Large Language Models](https://mlsanity.com/paper/2609.14168/) (2026)

## Abstract (arXiv)

Knowledge-intensive language understanding tasks require Language Models (LMs) to integrate relevant context, mitigating their inherent weaknesses, such as incomplete or outdated knowledge. However, conflicting knowledge can be present in the LM's parameters, termed intra-memory conflict, which can affect a model's propensity to accept contextual knowledge. To study the effect of intra-memory conflict on an LM's ability to accept relevant context, we utilize two knowledge conflict measures and a novel dataset containing inherently conflicting data, DynamicQA. This dataset includes facts with a temporal dynamic nature where facts can change over time and disputable dynamic facts, which can change depending on the viewpoint. DynamicQA is the first to include real-world knowledge conflicts and provide context to study the link between the different types of knowledge conflicts. We also evaluate several measures on their ability to reflect the presence of intra-memory conflict: semantic entropy and a novel coherent persuasion score. With our extensive experiments, we verify that LMs exhibit a greater degree of intra-memory conflict with dynamic facts compared to facts that have a single truth value. Furthermore, we reveal that facts with intra-memory conflict are harder to update with context, suggesting that retrieval-augmented generation will struggle with the most commonly adapted facts.