# LLM Circuit Analyses Are Consistent Across Training and Scale

arXiv: https://arxiv.org/abs/2407.10827 · HTML: https://mlsanity.com/paper/2407.10827/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 73 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Builds on (most cited first, 30 of 55)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Attribution Patching Outperforms Automated Circuit Discovery](https://mlsanity.com/paper/2310.10348/) (2023)
- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://mlsanity.com/paper/2305.00586/) (2023)
- [Localizing Model Behavior with Path Patching](https://mlsanity.com/paper/2304.05969/) (2023)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla](https://mlsanity.com/paper/2307.09458/) (2023)

## Built on by (most cited first, 30 of 63)

- [Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient](https://mlsanity.com/paper/2410.02984/) (2024)
- [PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs](https://mlsanity.com/paper/2503.09543/) (2025)
- [Mechanistic?](https://mlsanity.com/paper/2410.09087/) (2024)
- [How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training](https://mlsanity.com/paper/2502.11196/) (2025)
- [Inferring Functionality of Attention Heads from their Parameters](https://mlsanity.com/paper/2412.11965/) (2024)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them](https://mlsanity.com/paper/2507.10616/) (2025)
- [Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Units](https://mlsanity.com/paper/2601.21996/) (2026)
- [Mechanistic evaluation of Transformers and state space models](https://mlsanity.com/paper/2505.15105/) (2025)
- [Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning](https://mlsanity.com/paper/2507.10624/) (2025)
- [Influence Dynamics and Stagewise Data Attribution](https://mlsanity.com/paper/2510.12071/) (2025)
- [Circuit Stability Characterizes Language Model Generalization](https://mlsanity.com/paper/2505.24731/) (2025)
- [Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining](https://mlsanity.com/paper/2509.05291/) (2025)
- [Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models](https://mlsanity.com/paper/2504.15471/) (2025)
- [FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models](https://mlsanity.com/paper/2505.20225/) (2025)
- [Toward a Theory of Generalizability in LLM Mechanistic Interpretability Research](https://mlsanity.com/paper/2509.22831/) (2025)
- [Investigating Representation Universality: Case Study on Genealogical Representations](https://mlsanity.com/paper/2410.08255/) (2024)
- [ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behavior](https://mlsanity.com/paper/2505.20076/) (2025)
- [Attribution-Guided Pruning for Insight and Control: Circuit Discovery and Targeted Correction in Small-scale LLMs](https://mlsanity.com/paper/2506.13727/) (2025)
- [Convergence and Divergence of Language Models under Different Random Seeds](https://mlsanity.com/paper/2509.26643/) (2025)
- [Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity](https://mlsanity.com/paper/2511.21974/) (2025)
- [Suppressing Final Layer Hidden State Jumps in Transformer Pretraining](https://mlsanity.com/paper/2601.18302/) (2026)
- [Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic](https://mlsanity.com/paper/2601.22510/) (2026)
- [Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedish](https://mlsanity.com/paper/2602.03484/) (2026)
- [Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer](https://mlsanity.com/paper/2602.19058/) (2026)
- [Differentiable Faithfulness Alignment for Cross-Model Circuit Transfer](https://mlsanity.com/paper/2604.24302/) (2026)
- [Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics](https://mlsanity.com/paper/2606.06533/) (2026)
- [The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own](https://mlsanity.com/paper/2503.05760/) (2025)
- [How important is language for human-like intelligence?](https://mlsanity.com/paper/2509.15560/) (2025)
- [Review of Hallucination Understanding in Large Language and Vision Models](https://mlsanity.com/paper/2510.00034/) (2025)

## Abstract (arXiv)

Most currently deployed large language models (LLMs) undergo continuous training or additional finetuning. By contrast, most research into LLMs' internal mechanisms focuses on models at one snapshot in time (the end of pre-training), raising the question of whether their results generalize to real-world settings. Existing studies of mechanisms over time focus on encoder-only or toy models, which differ significantly from most deployed models. In this study, we track how model mechanisms, operationalized as circuits, emerge and evolve across 300 billion tokens of training in decoder-only LLMs, in models ranging from 70 million to 2.8 billion parameters. We find that task abilities and the functional components that support them emerge consistently at similar token counts across scale. Moreover, although such components may be implemented by different attention heads over time, the overarching algorithm that they implement remains. Surprisingly, both these algorithms and the types of components involved therein can replicate across model scale. These results suggest that circuit analyses conducted on small models at the end of pre-training can provide insights that still apply after additional pre-training and over model scale.