# LMs as Task-Specific Knowledge Bases: An Interpretability Analysis

arXiv: https://arxiv.org/abs/2606.27237 · HTML: https://mlsanity.com/paper/2606.27237/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons](https://mlsanity.com/paper/2308.13198/) (2023)
- [Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs](https://mlsanity.com/paper/2603.09906/) (2026)
- [Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts](https://mlsanity.com/paper/2412.04614/) (2024)
- [Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality](https://mlsanity.com/paper/2602.14080/) (2026)
- [Tracing Multilingual Factual Knowledge Acquisition in Pretraining](https://mlsanity.com/paper/2505.14824/) (2025)
- [Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency](https://mlsanity.com/paper/2603.17102/) (2026)
- [Where Knowledge Collides: A Mechanistic Study of Intra-Memory Knowledge Conflict in Language Models](https://mlsanity.com/paper/2601.09445/) (2026)
- [LLM Unlearning Should Be Form-Independent](https://mlsanity.com/paper/2506.07795/) (2025)
- [Friends and Grandmothers in Silico: Localizing Entity Cells in Language Models](https://mlsanity.com/paper/2604.01404/) (2026)

## Abstract (arXiv)

Language models (LMs) capture large amounts of factual knowledge applicable to a wide range of tasks, motivating the view of their parameters as a knowledge base. An important property of knowledge bases is that different queries for the same fact return consistent results, drawing on a single source of truth. We investigate whether LMs satisfy this property through behavioral and mechanistic analyses. Our results suggest that they encode knowledge in a task-specific manner. Behaviorally, facts acquired on one task frequently fail to co-emerge on others during training. Parameter localization experiments suggest a mechanistic explanation, revealing distinct parameter subsets underlying different tasks for the same fact. Finally, we show that chain-of-thought reasoning draws part of its effectiveness from engaging task-specific parameters beyond those tied to the evaluation task. Our findings suggest that what the model knows and how it is asked are intertwined in parameter space, undermining the "knowledge base" analogy and carrying implications for the reliability and controllability of factual knowledge in LMs.