# How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training

arXiv: https://arxiv.org/abs/2502.11196 · HTML: https://mlsanity.com/paper/2502.11196/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 58 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [Attribution Patching Outperforms Automated Circuit Discovery](https://mlsanity.com/paper/2310.10348/) (2023)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](https://mlsanity.com/paper/2312.05934/) (2023)
- [Towards Continual Knowledge Learning of Language Models](https://mlsanity.com/paper/2110.03215/) (2021)
- [Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms](https://mlsanity.com/paper/2403.17806/) (2024)
- [Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws](https://mlsanity.com/paper/2404.05405/) (2024)
- [Physics of Language Models: Part 3.2, Knowledge Manipulation](https://mlsanity.com/paper/2309.14402/) (2023)
- [Circuit Component Reuse Across Tasks in Transformer Language Models](https://mlsanity.com/paper/2310.08744/) (2023)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)
- [Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap](https://mlsanity.com/paper/2401.10034/) (2024)
- [How Do Large Language Models Acquire Factual Knowledge During Pretraining?](https://mlsanity.com/paper/2406.11813/) (2024)
- [Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models](https://mlsanity.com/paper/2407.07263/) (2024)

## Built on by (most cited first, 23 of 23)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [How do language models learn facts? Dynamics, curricula and hallucinations](https://mlsanity.com/paper/2503.21676/) (2025)
- [Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training](https://mlsanity.com/paper/2509.25758/) (2025)
- [Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Models](https://mlsanity.com/paper/2506.18141/) (2025)
- [Discursive Circuits: How Do Language Models Understand Discourse Relations?](https://mlsanity.com/paper/2510.11210/) (2025)
- [Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models](https://mlsanity.com/paper/2601.08383/) (2026)
- [Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis](https://mlsanity.com/paper/2505.14406/) (2025)
- [Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models](https://mlsanity.com/paper/2510.06107/) (2025)
- [What Does Loss Optimization Actually Teach, If Anything? Knowledge Dynamics in Continual Pre-training of LLMs](https://mlsanity.com/paper/2601.03858/) (2026)
- [From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space](https://mlsanity.com/paper/2604.14142/) (2026)
- [Programming by Backprop: An Instruction is Worth 100 Examples When Finetuning LLMs](https://mlsanity.com/paper/2506.18777/) (2025)
- [Progressive Homeostatic and Plastic Prompt Tuning for Audio-Visual Multi-Task Incremental Learning](https://mlsanity.com/paper/2507.21588/) (2025)
- [A Review of Developmental Interpretability in Large Language Models](https://mlsanity.com/paper/2508.15841/) (2025)
- [ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attack](https://mlsanity.com/paper/2509.25843/) (2025)
- [How Do Large Language Models Learn Concepts During Continual Pre-Training?](https://mlsanity.com/paper/2601.03570/) (2026)
- [Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining](https://mlsanity.com/paper/2601.22108/) (2026)
- [Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models](https://mlsanity.com/paper/2601.14758/) (2026)
- [V-DyKnow: A Dynamic Benchmark for Time-Sensitive Knowledge in Vision Language Models](https://mlsanity.com/paper/2603.16581/) (2026)
- [Inside-Out: Measuring Generalization in Vision Transformers Through Inner Workings](https://mlsanity.com/paper/2604.08192/) (2026)
- [DECSELFMASK: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classification](https://mlsanity.com/paper/2606.09466/) (2026)
- [Benchmarking Knowledge Editing using Logical Rules](https://mlsanity.com/paper/2606.10554/) (2026)
- [A theoretical model for task routing in mixture-of-expert transformers](https://mlsanity.com/paper/2606.14398/) (2026)
- [Identifying Temporal Features within Transcoders for Time Sensitive Factual Recall](https://mlsanity.com/paper/2609.33183/) (2026)

## Abstract (arXiv)

Despite exceptional capabilities in knowledge-intensive tasks, Large Language Models (LLMs) face a critical gap in understanding how they internalize new knowledge, particularly how to structurally embed acquired knowledge in their neural computations. We address this issue through the lens of knowledge circuit evolution, identifying computational subgraphs that facilitate knowledge storage and processing. Our systematic analysis of circuit evolution throughout continual pre-training reveals several key findings: (1) the acquisition of new knowledge is influenced by its relevance to pre-existing knowledge; (2) the evolution of knowledge circuits exhibits a distinct phase shift from formation to optimization; (3) the evolution of knowledge circuits follows a deep-to-shallow pattern. These insights not only advance our theoretical understanding of the mechanisms of new knowledge acquisition in LLMs, but also provide potential implications for improving continual pre-training strategies to enhance model performance. Code and data will be available at https://github.com/zjunlp/DynamicKnowledgeCircuits.