# Knowledge Localization: Mission Not Accomplished? Enter Query Localization!

arXiv: https://arxiv.org/abs/2405.14117 · HTML: https://mlsanity.com/paper/2405.14117/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons](https://mlsanity.com/paper/2404.02431/) (2024)
- [A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions](https://mlsanity.com/paper/2202.11912/) (2022)
- [Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark](https://mlsanity.com/paper/2305.17553/) (2023)
- [Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons](https://mlsanity.com/paper/2308.13198/) (2023)
- [Sequential Integrated Gradients: a simple but effective method for explaining language models](https://mlsanity.com/paper/2305.15853/) (2023)
- [What does the Knowledge Neuron Thesis Have to do with Knowledge?](https://mlsanity.com/paper/2405.02421/) (2024)
- [Identifying Semantic Induction Heads to Understand In-Context Learning](https://mlsanity.com/paper/2402.13055/) (2024)
- [Discretized Integrated Gradients for Explaining Language Models](https://mlsanity.com/paper/2108.13654/) (2021)
- [Unveiling the Pitfalls of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2310.02129/) (2023)
- [Emptying the Ocean with a Spoon: Should We Edit Models?](https://mlsanity.com/paper/2310.11958/) (2023)
- [Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks](https://mlsanity.com/paper/2401.17585/) (2024)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Investigating Saturation Effects in Integrated Gradients](https://mlsanity.com/paper/2010.12697/) (2020)

## Built on by (most cited first, 13 of 13)

- [Can Knowledge Editing Really Correct Hallucinations?](https://mlsanity.com/paper/2410.16251/) (2024)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Models](https://mlsanity.com/paper/2407.01920/) (2024)
- [Can Editing LLMs Inject Harm?](https://mlsanity.com/paper/2407.20224/) (2024)
- [Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer](https://mlsanity.com/paper/2506.01115/) (2025)
- [Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Models](https://mlsanity.com/paper/2409.00617/) (2024)
- [Fine-tuning Done Right in Model Editing](https://mlsanity.com/paper/2509.22072/) (2025)
- [The Knowledge Microscope: Features as Better Analytical Lenses than Neurons](https://mlsanity.com/paper/2502.12483/) (2025)
- [Revealing the Deceptiveness of Knowledge Editing: A Mechanistic Analysis of Superficial Editing](https://mlsanity.com/paper/2505.12636/) (2025)
- [One Mind, Many Tongues: A Deep Dive into Language-Agnostic Knowledge Neurons in Large Language Models](https://mlsanity.com/paper/2411.17401/) (2024)
- [Beyond Black-Box Interventions: Latent Probing for Faithful Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.12460/) (2025)
- [Know-MRI: A Knowledge Mechanisms Revealer&Interpreter for Large Language Models](https://mlsanity.com/paper/2506.08427/) (2025)
- [Storage Is Not Strategy: State-Conditioned Support Control for LLM Unlearning](https://mlsanity.com/paper/2609.37858/) (2026)

## Abstract (arXiv)

Large language models (LLMs) store extensive factual knowledge, but the mechanisms behind how they store and express this knowledge remain unclear. The Knowledge Neuron (KN) thesis is a prominent theory for explaining these mechanisms. This theory is based on the Knowledge Localization (KL) assumption, which suggests that a fact can be localized to a few knowledge storage units, namely knowledge neurons. However, this assumption has two limitations: first, it may be too rigid regarding knowledge storage, and second, it neglects the role of the attention module in knowledge expression. In this paper, we first re-examine the KL assumption and demonstrate that its limitations do indeed exist. To address these, we then present two new findings, each targeting one of the limitations: one focusing on knowledge storage and the other on knowledge expression. We summarize these findings as Query Localization (QL) assumption and argue that the KL assumption can be viewed as a simplification of the QL assumption. Based on QL assumption, we further propose the Consistency-Aware KN modification method, which improves the performance of knowledge modification, further validating our new assumption. We conduct 39 sets of experiments, along with additional visualization experiments, to rigorously confirm our conclusions. Code is available at https://github.com/heng840/KnowledgeLocalization.