mlsanityData to 14 Oct 2026

An Adversarial Example for Direct Logit Attribution: Memory Management in GELU-4L

arXiv 2310.07325 ↗First posted 11 Oct 2023cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Prior work suggests that language models manage the limited bandwidth of the residual stream through a "memory management" mechanism, where certain attention heads and MLP layers clear residual stream directions set by earlier layers. Our study provides concrete evidence for this erasure phenomenon in a 4-layer transformer, identifying heads that consistently remove the output of earlier heads. We further demonstrate that direct logit attribution (DLA), a common technique for interpreting the output of intermediate transformer layers, can show misleading results by not accounting for erasure.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 20 of its 39 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

20 of 39 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  8. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  9. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  10. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  11. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
8 more
  1. 2013Direct and Indirect Effectscited by 280
  2. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  3. 2023Localizing Model Behavior with Path Patchingcited by 195
  4. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  5. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156
  6. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  7. 2022Natural Language Descriptions of Deep Visual Featurescited by 137
  8. 2022Analyzing Transformers in Embedding Spacecited by 132

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 6.

11 papers.

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  2. 2024Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMscited by 42
  3. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 31
  4. 2024Optimal ablation for interpretabilitycited by 25
  5. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  6. 2026Discovering Universal Activation Directions for PII Leakage in Language Modelscited by 1
  7. 2026The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representationscited by 1
  8. 2026Through the Looking Glass: Directly Reading and Writing Transformerscited by 1
  9. 2025Evaluation Framework for Highlight Explanations of Context Utilisation in Language Modelscited by 0
  10. 2026RACE: Scalable Statistical Estimation of Functional Consistency in LLM Neuronscited by 0
  11. 2026Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influencecited by 0