# Copying Before Suppression: What Drives a Below-Chance Dip During Language Model Training?

arXiv: https://arxiv.org/abs/2610.04119 · HTML: https://mlsanity.com/paper/2610.04119/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 14 of 14)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [How to use and interpret activation patching](https://mlsanity.com/paper/2404.15255/) (2024)
- [Localizing Model Behavior with Path Patching](https://mlsanity.com/paper/2304.05969/) (2023)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [LLM Circuit Analyses Are Consistent Across Training and Scale](https://mlsanity.com/paper/2407.10827/) (2024)
- [PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs](https://mlsanity.com/paper/2503.09543/) (2025)

## Abstract (arXiv)

Mechanistic interpretability usually studies fully trained models, yet the computations that drive a behaviour can change while the model is still learning the task. On the Indirect Object Identification task, a model should continue with the name mentioned once rather than the name mentioned twice. Pythia models pass through an early training window in which they prefer the repeated name, so accuracy in a choice between the two names falls below one half while language-model loss on a fixed text sample keeps decreasing across the same window. The window reflects a temporary imbalance between two computations. We identify one cause of the wrong preference by selecting a set of attention heads that write the repeated name, on prompts separate from those used for causal evaluation, keeping that selection fixed, and then replacing each head's final-token output with its average output on a separate set of non-repeated-name prompts. This improves the correct-minus-repeated logit difference in a separately trained 160M model and in the official 160M, 410M, and 1B models. At 160M, the head that lowers the repeated name in the mature model shows little of its mature behaviour at this point. It directs less than one percent of its attention to the repeated mention, and its output makes almost no direct contribution to lowering that name's logit. Both properties grow over the interval in which behaviour recovers. Across the 160M, 410M, and 1B models, transplanting the corresponding head's mature parameters into the early checkpoint recovers 35 to 68 percent of the total improvement in the correct-minus-repeated logit difference seen by the end of training. Related early-to-late reversals appear at further Pythia scales, in two independently trained GPT-2 models, and in OLMo. A mature circuit can therefore conceal a transient causal configuration that shaped behaviour earlier in training.