mlsanityData to 7 Oct 2026

Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents

arXiv 2404.16032 ↗First posted 24 Apr 2024cs.LGcited by 27

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Retrieval-augmented generation (RAG) mitigates many problems of fully parametric language models, such as temporal degradation, hallucinations, and lack of grounding. In RAG, the model's knowledge can be updated from documents provided in context. This leads to cases of conflict between the model's parametric knowledge and the contextual information, where the model may not always update its knowledge. Previous work studied context-memory knowledge conflicts by creating synthetic documents that contradict the model's correct parametric answers. We present a framework for studying such knowledge conflicts in a realistic setup. We update incorrect parametric knowledge using real conflicting documents. This reflects how knowledge conflicts arise in practice. In this realistic scenario, we find that knowledge updates fail less often than previously reported. In cases where the models still fail to update their answers, we find a parametric bias: the incorrect parametric answer appearing in context makes the knowledge update likelier to fail. These results suggest that the factual parametric knowledge of LLMs can negatively influence their reading abilities and behaviors. Our code is available at https://github.com/kortukov/realistic_knowledge_conflicts/ .

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 38 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

31 of 38 references.

References · 31

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
19 more
  1. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  2. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  3. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  4. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  5. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  6. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  7. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  8. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  9. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  10. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  11. 2022Prompting GPT-3 To Be Reliablecited by 258
  12. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  13. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  14. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  15. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
  16. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  17. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  18. 2024LexDrafter: Terminology Drafting for Legislative Documents using Retrieval Augmented Generationcited by 6
  19. 2023Comparable Demonstrations are Important in In-Context Learning: A Novel Perspective on Demonstration Selectioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 12.

27 papers.

Sort:

By area:

  1. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  2. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievalscited by 27
  3. 2025DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMscited by 21
  4. 2024Retrieval-Augmented Generation with Estimation of Source Reliabilitycited by 19
  5. 2025Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflictcited by 15
  6. 2025Do RAG Systems Really Suffer From Positional Bias?cited by 12
  7. 2025ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generationcited by 11
  8. 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
  9. 2024A Reality Check on Context Utilisation for Retrieval-Augmented Generationcited by 9
  10. 2025"Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Modelscited by 5
  11. 2025Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Modelscited by 4
  12. 2025KScope: A Framework for Characterizing the Knowledge Status of Language Modelscited by 3
15 more
  1. 2025How Training Data Shapes the Use of Parametric and In-Context Knowledge in Language Modelscited by 3
  2. 2025When Facts Change: Probing LLMs on Evolving Knowledge with evolveQAcited by 2
  3. 2026When Should Models Change Their Minds? Contextual Belief Management in Large Language Modelscited by 2
  4. 2026Evaluating Evidence Grounding Under User Pressure in Instruction-Tuned Language Modelscited by 1
  5. 2026Information Discernment in Large Language Modelscited by 1
  6. 2026Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflictscited by 1
  7. 2025CCSK:Cognitive Convection of Self-Knowledge Based Retrieval Augmentation for Large Language Modelscited by 0
  8. 2026CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Lawcited by 0
  9. 2026The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generationcited by 0
  10. 2026Persistent Priors, Preserved Targets: A Stroop-Style Paradigm for Lexical Overridecited by 0
  11. 2026When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Modelscited by 0
  12. 2026Shortcut Before Circuit: Document Statistics Time In-Context Conflict Resolutioncited by 0
  13. 2026Rewired or Gated? How Instruction Tuning Shapes Knowledge-Conflict Circuits in LLMscited by 0
  14. 2026Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answerscited by 0
  15. 2026What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Usecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Models
  2. 2024Deciphering the Interplay of Parametric and Non-parametric Memory in Retrieval-augmented Language Models
  3. 2025Resisting Contextual Interference in RAG via Parametric-Knowledge Reinforcement
  4. 2025FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation
  5. 2024From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries
  6. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence
  7. 2025Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancement
  8. 2025Conflict-Aware Soft Prompting for Retrieval-Augmented Generation