mlsanityData to 14 Oct 2026

LLM Layers Immediately Correct Each Other

arXiv 2609.07876 ↗First posted 7 Sep 2026cs.CL cs.LGcited by 3

On the map: Mechanistic interpretability → Knowledge and representations inside LLMs

Abstract (arXiv)

Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linear, semantically meaningful features. Such methods are commonly interpreted as identifying features that persist in the residual stream and that subsequent layers build upon. We challenge this view by identifying the Transformer Layer Correction Mechanism (TLCM), wherein adjacent transformer layers systematically counteract portions of each other's contributions. TLCM appears in 5 out of 7 major open-source model families and activates across nearly all tokens in diverse texts. We show that TLCM emerges during pretraining, operates most strongly on contextually dependent tokens, and adaptively calibrates its correction strength based on the preceding layer's output. Using the layer Jacobian, we further show that TLCM selectively corrects specific subspaces while reinforcing others, which we interpret through a ``propose-and-reject'' framework in which layers propose candidate features and subsequent layers selectively remove inappropriate ones. This dynamic suggests that the residual stream at any layer contains transient proposals alongside persistent features, helping explain why SAE feature descriptions often have low specificity, why effective model steering requires extreme feature amplification, and why transcoders hold a theoretical advantage over SAEs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability. Builds on 30 of its 46 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

30 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lens14 Mar

Other references · 29

  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  8. 2024Scaling and evaluating sparse autoencoderscited by 553
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  10. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  11. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  12. 2023Function Vectors in Large Language Modelscited by 308
17 more
  1. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  2. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  3. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  4. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  5. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
  6. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  7. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156
  8. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  9. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  10. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 92
  11. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 62
  12. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 51
  13. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 43
  14. 2024Explorations of Self-Repair in Language Modelscited by 39
  15. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  16. 2025Transcoders Beat Sparse Autoencoders for Interpretabilitycited by 25
  17. 2024Uncovering Uncertainty in Transformer Inferencecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Why Does Post-Training Quantization Work?cited by 1
  2. 2026Script Choice in LLMs: Evidence for Late-Layer Commitmentcited by 1
  3. 2026Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models
  2. 2025Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
  3. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similarity
  4. 2025Spectral Insights into Data-Oblivious Critical Layers in Large Language Models
  5. 2026Evidence for feature-specific error correction in LLMs
  6. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
  7. 2025Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability
  8. 2024Understanding Layer Significance in LLM Alignment