mlsanityData to 14 Oct 2026

Knowledge Circuits in Pretrained Transformers

arXiv 2405.17969 ↗First posted 28 May 2024cs.CL cs.AI cs.CVcited by 60

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

The remarkable capabilities of modern large language models are rooted in their vast repositories of knowledge encoded within their parameters, enabling them to perceive the world and engage in reasoning. The inner workings of how these models store knowledge have long been a subject of intense interest and investigation among researchers. To date, most studies have concentrated on isolated components within these models, such as the Multilayer Perceptrons and attention head. In this paper, we delve into the computation graph of the language model to uncover the knowledge circuits that are instrumental in articulating specific knowledge. The experiments, conducted with GPT2 and TinyLLAMA, have allowed us to observe how certain information heads, relation heads, and Multilayer Perceptrons collaboratively encode knowledge within the model. Moreover, we evaluate the impact of current knowledge editing techniques on these knowledge circuits, providing deeper insights into the functioning and constraints of these editing methodologies. Finally, we utilize knowledge circuits to analyze and interpret language model behaviors such as hallucinations and in-context learning. We believe the knowledge circuits hold potential for advancing our understanding of Transformers and guiding the improved design of knowledge editing. Code and data are available in https://github.com/zjunlp/KnowledgeCircuits.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 56 of its 77 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

56 of 77 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT19 Feb
  2. 2023Attribution Patching Outperforms Automated Circuit Discovery16 Oct
  3. 2023Circuit Component Reuse Across Tasks in Transformer Language Models12 Oct
  4. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability28 Apr
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 51

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  8. 2022Mass-Editing Memory in a Transformercited by 1,001
  9. 2022Toy Models of Superpositioncited by 971
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  12. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
38 more
  1. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  2. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  3. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  4. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  5. 2023Function Vectors in Large Language Modelscited by 308
  6. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  7. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  8. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  9. 2013Direct and Indirect Effectscited by 280
  10. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  11. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
  12. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  13. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  14. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 215
  15. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  16. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  17. 2023Localizing Model Behavior with Path Patchingcited by 195
  18. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  19. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  20. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  21. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 98
  22. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 98
  23. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 95
  24. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 91
  25. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  26. 2024Unified Hallucination Detection for Multimodal Large Language Modelscited by 75
  27. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  28. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  29. 2024How to think step-by-step: A mechanistic understanding of chain-of-thought reasoningcited by 53
  30. 2024What does the Knowledge Neuron Thesis Have to do with Knowledge?cited by 49
  31. 2024Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMscited by 42
  32. 2023Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operationcited by 33
  33. 2023VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformerscited by 32
  34. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 31
  35. 2023Circuit Breaking: Removing Model Behaviors with Targeted Ablationcited by 27
  36. 2023Memory Injections: Correcting Multi-Hop Reasoning Failures during Inference in Transformer-Based Language Modelscited by 21
  37. 2023Discovering Knowledge-Critical Subnetworks in Pretrained Language Modelscited by 16
  38. 2024Investigating Multi-Hop Factual Shortcuts in Knowledge Editing of Large Language Modelscited by 14

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 29; 2026: 24.

60 papers.

Sort:

By area:

  1. 2024Attention Heads of Large Language Models: A Surveycited by 67
  2. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyondcited by 61
  3. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  4. 2024Can Knowledge Editing Really Correct Hallucinations?cited by 28
  5. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  6. 2025How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Trainingcited by 23
  7. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 16
  8. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformercited by 12
  9. 2025Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoningcited by 11
  10. 2026Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guaranteescited by 11
  11. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generationcited by 10
  12. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoningcited by 10
38 more
  1. 2025Rethinking Circuit Completeness in Language Models: AND, OR, and ADDER Gatescited by 10
  2. 2026Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Modelscited by 10
  3. 2025Revealing and Mitigating Over-Attention in Knowledge Editingcited by 8
  4. 2025Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probingcited by 8
  5. 2025Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Trainingcited by 8
  6. 2025Unraveling LLM Jailbreaks Through Safety Knowledge Neuronscited by 7
  7. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 7
  8. 2025Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Modelscited by 6
  9. 2026Contextual Agentic Memory is a Memo, Not True Memorycited by 6
  10. 2026When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformerscited by 6
  11. 2026Circular Reasoning: Understanding Self-Reinforcing Loops in Large Reasoning Modelscited by 5
  12. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
  13. 2025LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgradescited by 4
  14. 2026SafeSeek: Universal Attribution of Safety Circuits in Language Modelscited by 4
  15. 2024CogSteer: Cognition-Inspired Selective Layer Intervention for Efficiently Steering Large Language Modelscited by 3
  16. 2025SetKE: Knowledge Editing for Knowledge Elements Overlapcited by 3
  17. 2025Understanding Post-Training Structural Changes in Large Language Modelscited by 3
  18. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Informationcited by 3
  19. 2025Steering Information Utility in Key-Value Memory for Language Model Post-Trainingcited by 2
  20. 2025Measuring Uncertainty in Transformer Circuits with Effective Information Consistencycited by 2
  21. 2025The Curious Case of Factual (Mis)Alignment between LLMs' Short- and Long-Form Answerscited by 2
  22. 2026Explaining Grokking in Transformers through the Lens of Inductive Biascited by 2
  23. 2026Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarkscited by 2
  24. 2026Demystifying Variance in Circuit Discovery of LLMscited by 2
  25. 2024Skill Path: Unveiling Language Skills from Circuit Graphscited by 1
  26. 2025Capability Localization: Capabilities Can be Localized rather than Individual Knowledgecited by 1
  27. 2025Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledgecited by 1
  28. 2025Identifying a Circuit for Verb Conjugation in GPT-2cited by 1
  29. 2025LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimizationcited by 1
  30. 2025When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Modelscited by 1
  31. 2026How Do Large Language Models Learn Concepts During Continual Pre-Training?cited by 1
  32. 2026Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshapingcited by 1
  33. 2026When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalitiescited by 1
  34. 2024Maintaining Informative Coherence: Migrating Hallucinations in Large Language Models via Absorbing Markov Chainscited by 0
  35. 2025Quantifying Logical Consistency in Transformers via Query-Key Alignmentcited by 0
  36. 2025CircuitProbe: Tracing Visual Temporal Evidence Flow in Video Language Modelscited by 0
  37. 2025Towards Atoms of Large Language Modelscited by 0
  38. 2025Programmatic Representation Learning with Language Modelscited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding Language Model Circuits through Knowledge Editing
  2. 2025CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners
  3. 2024Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference
  4. 2026Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized Representations
  5. 2023MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models
  6. 2026Addressing the Reasoning Gap: Mechanistic Circuit-Based Knowledge Editing in Large Language Models
  7. 2025Identifying Multi-modal Knowledge Neurons in Pretrained Transformers via Two-stage Filtering
  8. 2025Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation Analysis