mlsanityData to 13 Oct 2026

When Context Leads but Parametric Memory Follows in Large Language Models

arXiv 2409.08435 ↗First posted 13 Sep 2024cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable progress in leveraging diverse knowledge sources. This study investigates how nine widely used LLMs allocate knowledge between local context and global parameters when answering open-ended questions in knowledge-consistent scenarios. We introduce a novel dataset, WikiAtomic, and systematically vary context sizes to analyze how LLMs prioritize and utilize the provided information and their parametric knowledge in knowledge-consistent scenarios. Additionally, we also study their tendency to hallucinate under varying context sizes. Our findings reveal consistent patterns across models, including a consistent reliance on both contextual (around 70%) and parametric (around 30%) knowledge, and a decrease in hallucinations with increasing context. These insights highlight the importance of more effective context organization and developing models that use input more deterministically for robust performance.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

25 of 37 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,412
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  5. 2019Language Models as Knowledge Bases?cited by 1,931
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,268
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,126
  8. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 384
  9. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 352
  10. 2024Knowledge Conflicts for LLMs: A Surveycited by 277
  11. 2022Prompting GPT-3 To Be Reliablecited by 259
  12. 2023Fine-tuning Language Models for Factualitycited by 219
13 more
  1. 2022Large Language Models with Controllable Working Memorycited by 169
  2. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  3. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  4. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  5. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 94
  6. 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
  7. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  8. 2024Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden Statescited by 52
  9. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  10. 2024Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problemcited by 42
  11. 2023Prompting Large Language Models for Counterfactual Generation: An Empirical Studycited by 36
  12. 2024Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Taskscited by 23
  13. 2024Does the Generator Mind its Contexts? An Analysis of Generative Model Faithfulness under Context Transfercited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2026Doc-to-LoRA: Learning to Instantly Internalize Contextscited by 39
  2. 2025Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancementcited by 25
  3. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  4. 2025Do Role-Playing Agents Practice What They Preach? Belief-Behavior Consistency in LLM-Based Simulations of Human Trustcited by 6
  5. 2025"Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Modelscited by 5
  6. 2025KScope: A Framework for Characterizing the Knowledge Status of Language Modelscited by 3
  7. 2025Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace Disentanglementcited by 1
  8. 2026Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMscited by 0
  9. 2026Evaluating the accuracy of KV cache reuse techniquescited by 0
  10. 2026When Upstream Messages Override Correct Answers: A Controlled Study of Multi-Agent LLM Collaborationcited by 0
  11. 2026Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinatecited by 0