# Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflict

arXiv: https://arxiv.org/abs/2506.06485 · HTML: https://mlsanity.com/paper/2506.06485/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Builds on (most cited first, 28 of 28)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence](https://mlsanity.com/paper/2210.13701/) (2022)
- [Rationale-Augmented Ensembles in Language Models](https://mlsanity.com/paper/2207.00747/) (2022)
- [Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2402.14409/) (2024)
- [Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2402.18154/) (2024)
- [JudgeLRM: Large Reasoning Models as a Judge](https://mlsanity.com/paper/2504.00050/) (2025)
- [What Evidence Do Language Models Find Convincing?](https://mlsanity.com/paper/2402.11782/) (2024)
- [WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia](https://mlsanity.com/paper/2406.13805/) (2024)
- [Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs](https://mlsanity.com/paper/2410.08145/) (2024)
- [AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge](https://mlsanity.com/paper/2409.07394/) (2024)
- [Taming Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2503.10996/) (2025)
- [Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents](https://mlsanity.com/paper/2404.16032/) (2024)
- [Intuitive or Dependent? Investigating LLMs' Behavior Style to Conflicting Prompts](https://mlsanity.com/paper/2309.17415/) (2023)
- [How Robust are LLMs to In-Context Majority Label Bias?](https://mlsanity.com/paper/2312.16549/) (2023)
- [Mitigating Misleading Chain-of-Thought Reasoning with Selective Filtering](https://mlsanity.com/paper/2403.19167/) (2024)

## Built on by (most cited first, 16 of 16)

- [Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence](https://mlsanity.com/paper/2601.11886/) (2026)
- [Context-CoT: Enhancing Context Learning via High-Quality Reasoning Synthesis](https://mlsanity.com/paper/2605.25354/) (2026)
- [Three Regimes of Context-Parametric Conflict: A Predictive Framework and Empirical Validation](https://mlsanity.com/paper/2605.11574/) (2026)
- [Injecting Falsehoods: Adversarial Man-in-the-Middle Attacks Undermining Factual Recall in LLMs](https://mlsanity.com/paper/2511.05919/) (2025)
- [CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios](https://mlsanity.com/paper/2605.07830/) (2026)
- [Diagnosing Retrieval Bias Under Multiple In-Context Knowledge Updates in Large Language Models](https://mlsanity.com/paper/2603.12271/) (2026)
- [OCC-RAG: Optimal Cognitive Core for Faithful Question Answering](https://mlsanity.com/paper/2606.00683/) (2026)
- [Persistent Priors, Preserved Targets: A Stroop-Style Paradigm for Lexical Override](https://mlsanity.com/paper/2606.07555/) (2026)
- [Prompts in the Wild: A Large Analyzed Collection of Transactional Prompts in Code](https://mlsanity.com/paper/2608.12905/) (2026)
- [Calibrating Criterion Revision in LLM Agents: Failure Modes and a Trace-Anchored Protocol](https://mlsanity.com/paper/2608.20729/) (2026)
- [Record Grouping Controls Evidence Weight in Language Models](https://mlsanity.com/paper/2609.08698/) (2026)
- [Do LLMs Make More Mistakes If They Do Not Believe the Input Data?](https://mlsanity.com/paper/2609.09363/) (2026)
- [MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors](https://mlsanity.com/paper/2609.27590/) (2026)
- [What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use](https://mlsanity.com/paper/2609.32991/) (2026)
- [Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities](https://mlsanity.com/paper/2610.00606/) (2026)
- [Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict](https://mlsanity.com/paper/2610.12360/) (2026)

## Abstract (arXiv)

Large language models (LLMs) draw on both contextual information and parametric memory, yet these sources can conflict. Prior studies have largely examined this issue in contextual question answering, implicitly assuming that tasks should rely on the provided context, leaving unclear how LLMs behave when tasks require different types and degrees of knowledge utilization. We address this gap with a model-agnostic diagnostic framework that holds underlying knowledge constant while introducing controlled conflicts across tasks with varying knowledge demands. Experiments on representative open-weight and proprietary LLMs show that performance degradation under conflict is driven by both task-specific knowledge reliance and conflict plausibility; that strategies such as rationales or context reiteration increase context reliance, helping context-only tasks but harming those requiring parametric knowledge; and that these effects bias model-based evaluation, calling into question the reliability of LLMs as judges. Overall, our findings reveal that context-memory conflict is inherently task-dependent and motivate task-aware approaches to balancing context and memory in LLM deployment and evaluation.