mlsanityData to 14 Oct 2026

Even Heads Fix Odd Errors: Mechanistic Discovery and Surgical Repair in Transformer Attention

arXiv 2508.19414 ↗First posted 26 Aug 2025cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We present a mechanistic case study of a format-dependent reasoning failure in Llama-3.1-8B-Instruct, where the model incorrectly judges "9.11" as larger than "9.8" in chat or Q&A formats, but answers correctly in simple format. Through systematic intervention, we discover transformers implement even/odd attention head specialization: even indexed heads handle numerical comparison, while odd heads serve incompatible functions. The bug requires exactly 8 even heads at Layer 10 for perfect repair. Any combination of 8+ even heads succeeds, while 7 or fewer completely fails, revealing sharp computational thresholds with perfect redundancy among the 16 even heads. SAE analysis reveals the mechanism: format representations separate (10% feature overlap at Layer 7), then re-entangle with different weightings (80% feature overlap at Layer 10), with specific features showing 1.5x amplification in failing formats. We achieve perfect repair using only 25% of attention heads and identify a 60% pattern replacement threshold, demonstrating that apparent full-module requirements hide sophisticated substructure with implications for interpretability and efficiency. All of our code is available at https://github.com/gussand/surgeon.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

12 of 17 references.

References · 12

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  6. 2023Steering Language Models With Activation Engineeringcited by 922
  7. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  8. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  9. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 143
  10. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 76
  11. 2024Language Models Encode Numbers Using Digit Representations in Base 10cited by 32
  12. 2025Large Language Models and Mathematical Reasoning Failurescited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Surgical Repair of Insecure Code Generation in LLMscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
  2. 2025Attention as Binding: A Vector-Symbolic Perspective on Transformer Reasoning
  3. 2026Safety-Utility Conflicts Are Not Global: Surgical Alignment via Head-Level Diagnosis
  4. 2025Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers
  5. 2026Quantifying LLM Attention-Head Stability: Implications for Circuit Universality
  6. 2025Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers
  7. 2026A model of errors in transformers
  8. 2025Mechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AI