mlsanityData to 14 Oct 2026

Probing Language Models on Their Knowledge Source

arXiv 2410.05817 ↗First posted 8 Oct 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) often encounter conflicts between their learned, internal (parametric knowledge, PK) and external knowledge provided during inference (contextual knowledge, CK). Understanding how LLMs models prioritize one knowledge source over the other remains a challenge. In this paper, we propose a novel probing framework to explore the mechanisms governing the selection between PK and CK in LLMs. Using controlled prompts designed to contradict the model's PK, we demonstrate that specific model activations are indicative of the knowledge source employed. We evaluate this framework on various LLMs of different sizes and demonstrate that mid-layer activations, particularly those related to relations in the input, are crucial in predicting knowledge source selection, paving the way for more reliable models capable of handling knowledge conflicts effectively.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

19 of 31 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  10. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  12. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
7 more
  1. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  2. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  3. 2023Do Large Language Models Know What They Don't Know?cited by 243
  4. 2022A Review on Language Models as Knowledge Basescited by 157
  5. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  6. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  7. 2023A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepediacited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 2.

11 papers.

  1. 2025How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencodingcited by 19
  2. 2025Language Models Can Predict Their Own Behaviorcited by 8
  3. 2025Probing the Difficulty Perception Mechanism of Large Language Modelscited by 8
  4. 2025RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Mergingcited by 5
  5. 2025Bi-directional Model Cascading with Proxy Confidencecited by 2
  6. 2025Context Copying Modulation: The Role of Entropy Neurons in Managing Parametric and Contextual Knowledge Conflictscited by 1
  7. 2026Probing for Knowledge Attribution in Large Language Modelscited by 1
  8. 2026Do LLMs have core beliefs?cited by 1
  9. 2025Does This Look Familiar to You? Knowledge Analysis via Model Internal Representationscited by 0
  10. 2025That's Deprecated! Understanding, Detecting, and Steering Knowledge Conflicts in Language Models for Code Generationcited by 0
  11. 2025Classification EM-PCA for clustering and embeddingcited by 0