mlsanityData to 7 Oct 2026

Copy Suppression: Comprehensively Understanding an Attention Head

arXiv 2310.04625 ↗First posted 6 Oct 2023cs.LG cs.AI cs.CLcited by 73

Not on the map. Why?

Abstract (arXiv)

We present a single attention head in GPT-2 Small that has one main role across the entire training distribution. If components in earlier layers predict a certain token, and this token appears earlier in the context, the head suppresses it: we call this copy suppression. Attention Head 10.7 (L10H7) suppresses naive copying behavior which improves overall model calibration. This explains why multiple prior works studying certain narrow tasks found negative heads that systematically favored the wrong answer. We uncover the mechanism that the Negative Heads use for copy suppression with weights-based evidence and are able to explain 76.9% of the impact of L10H7 in GPT-2 Small. To the best of our knowledge, this is the most comprehensive description of the complete role of a component in a language model to date. One major effect of copy suppression is its role in self-repair. Self-repair refers to how ablating crucial model components results in downstream neural network parts compensating for this ablation. Copy suppression leads to self-repair: if an initial overconfident copier is ablated, then there is nothing to suppress. We show that self-repair is implemented by several mechanisms, one of which is copy suppression, which explains 39% of the behavior in a narrow task. Interactive visualisations of the copy suppression phenomena may be seen at our web app https://copy-suppression.streamlit.app/

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 20 of its 41 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

20 of 41 references.

References · 20

  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  2. 2022In-context Learning and Induction Headscited by 930
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  4. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  5. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  6. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  8. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  9. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  10. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  11. 2023Localizing Model Behavior with Path Patchingcited by 182
  12. 2020Compositional Explanations of Neuronscited by 172
8 more
  1. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  2. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  3. 2022Analyzing Transformers in Embedding Spacecited by 131
  4. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  5. 2021An Interpretability Illusion for BERTcited by 94
  6. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  7. 2016Highway and Residual Networks learn Unrolled Iterative Estimationcited by 52
  8. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 31; 2025: 19; 2026: 21.

73 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024How to use and interpret activation patchingcited by 191
  3. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  4. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  5. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  6. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  7. 2024Are self-explanations from Large Language Models faithful?cited by 101
  8. 2024Universal Neurons in GPT2 Language Modelscited by 100
  9. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  10. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 100
  11. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  12. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
38 more
  1. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  2. 2024Attention Heads of Large Language Models: A Surveycited by 65
  3. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  4. 2024LoFiT: Localized Fine-tuning on LLM Representationscited by 56
  5. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  6. 2024Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactualscited by 40
  7. 2024Extending Activation Steering to Broad Skills and Multiple Behaviourscited by 36
  8. 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runscited by 33
  9. 2025Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-rankingcited by 30
  10. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  11. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  12. 2025Taming Knowledge Conflicts in Language Modelscited by 27
  13. 2024Optimal ablation for interpretabilitycited by 21
  14. 2025Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysiscited by 20
  15. 2025Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Trackingcited by 17
  16. 2024Linking In-context Learning in Transformers to Human Episodic Memorycited by 15
  17. 2024Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Modelscited by 13
  18. 2024Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neurosciencecited by 11
  19. 2024You can remove GPT2's LayerNorm by fine-tuningcited by 11
  20. 2025Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?cited by 11
  21. 2025Towards Understanding the Nature of Attention with Low-Rank Sparse Decompositioncited by 9
  22. 2025Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretabilitycited by 7
  23. 2024When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Modelscited by 6
  24. 2026The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Researchcited by 6
  25. 2024Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neurosciencecited by 5
  26. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Regioncited by 5
  27. 2023Forbidden Facts: An Investigation of Competing Objectives in Llama-2cited by 4
  28. 2024Interpretability in Action: Exploratory Analysis of VPT, a Minecraft Agentcited by 4
  29. 2025Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagationcited by 4
  30. 2026Bi-Orthogonal Factor Decomposition for Vision Transformerscited by 4
  31. 2026Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Modelscited by 4
  32. 2025Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identificationcited by 3
  33. 2025Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insightcited by 3
  34. 2025Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learningcited by 2
  35. 2026Compressed Sensing for Capability Localization in Large Language Modelscited by 2
  36. 2026How LLMs Are Persuaded: A Few Attention Heads, Reroutedcited by 2
  37. 2026Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretabilitycited by 2
  38. 2026Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Failcited by 2

50 most cited of 73 All 73 (JSON)