mlsanityData to 14 Oct 2026

Neuron-Level Knowledge Attribution in Large Language Models

arXiv 2312.12141 ↗First posted 19 Dec 2023cs.CL cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Identifying important neurons for final predictions is essential for understanding the mechanisms of large language models. Due to computational constraints, current attribution techniques struggle to operate at neuron level. In this paper, we propose a static method for pinpointing significant neurons. Compared to seven other methods, our approach demonstrates superior performance across three metrics. Additionally, since most static methods typically only identify "value neurons" directly contributing to the final prediction, we propose a method for identifying "query neurons" which activate these "value neurons". Finally, we apply our methods to analyze six types of knowledge across both attention and feed-forward network (FFN) layers. Our method and analysis are helpful for understanding the mechanisms of knowledge storage and set the stage for future research in knowledge editing. The code is available on https://github.com/zepingyu0512/neuron-attribution.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

39 of 57 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  10. 2022Toy Models of Superpositioncited by 971
  11. 2019Attention is not Explanationcited by 929
  12. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
27 more
  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  2. 2019Attention is not not Explanationcited by 636
  3. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  4. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  5. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  6. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  7. 2019Is Attention Interpretable?cited by 389
  8. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  9. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  11. 2017The (Un)reliability of saliency methodscited by 333
  12. 2023Explainability for Large Language Models: A Surveycited by 333
  13. 2013Direct and Indirect Effectscited by 280
  14. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  15. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  16. 2023Localizing Model Behavior with Path Patchingcited by 195
  17. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  18. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
  19. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  20. 2022A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributionscited by 64
  21. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  22. 2020Towards Transparent and Explainable Attention Modelscited by 47
  23. 2020Why Attentions May Not Be Interpretable?cited by 41
  24. 2023Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Mapscited by 35
  25. 2021Attention Flows are Shapley Value Explanationscited by 30
  26. 2020Investigating Saturation Effects in Integrated Gradientscited by 22
  27. 2021VisQA: X-raying Vision and Language Reasoning in Transformerscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 22; 2026: 12.

37 papers.

Sort:

By area:

  1. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  2. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 43
  3. 2025Parametric Retrieval Augmented Generationcited by 38
  4. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 31
  5. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  6. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  7. 2025Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signalscited by 13
  8. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answeringcited by 11
  9. 2025ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generationcited by 11
  10. 2025Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMscited by 7
  11. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 7
  12. 2025UMoE: Unifying Attention and FFN with Shared Expertscited by 6
25 more
  1. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formatscited by 6
  2. 2025Brain-Inspired Exploration of Functional Networks and Key Neurons in Large Language Modelscited by 5
  3. 2025Representation Consistency for Accurate and Coherent LLM Answer Aggregationcited by 5
  4. 2026Universal Redundancies in Time Series Foundation Modelscited by 5
  5. 2025Dynamic and Parametric Retrieval-Augmented Generationcited by 3
  6. 2025Understanding Post-Training Structural Changes in Large Language Modelscited by 3
  7. 2025ACE: Attribution-Controlled Knowledge Editing for Multi-hop Factual Recallcited by 3
  8. 2025Truth Neuronscited by 2
  9. 2026Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigationcited by 2
  10. 2026Neuron-Level Emotion Control in Speech-Generative Large Audio-Language Modelscited by 2
  11. 2026Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Rewardcited by 2
  12. 2025ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Modelscited by 1
  13. 2025Neuron-Level Analysis of Cultural Understanding in Large Language Modelscited by 1
  14. 2025InterpDetect: Interpretable Signals for Detecting Hallucinations in Retrieval-Augmented Generationcited by 1
  15. 2026Neuron-Aware Data Selection In Instruction Tuning For Large Language Modelscited by 1
  16. 2025Time Course MechInterp: Analyzing the Evolution of Components and Knowledge in Large Language Modelscited by 0
  17. 2025GPT and Prejudice: A Sparse Approach to Understanding Learned Representations in Large Language Modelscited by 0
  18. 2025LLM-Augmented Symbolic NLU System for More Reliable Continuous Causal Statement Interpretationcited by 0
  19. 2025Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contextscited by 0
  20. 2026Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancementcited by 0
  21. 2026Mechanistic Decoding of Cognitive Constructs in Large Language Modelscited by 0
  22. 2026MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Modelscited by 0
  23. 2026Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Modelscited by 0
  24. 2026SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?cited by 0
  25. 2026Causal Routing for Unlearningcited by 0