mlsanityData to 14 Oct 2026

Slaying the Hydra: Interaction-Aware Circuit Discovery in Language Models

arXiv 2610.04017 ↗First posted 2 Oct 2026cs.LG cs.AI cs.CLcited by 0

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Localizing behavior to individual components of a language model is a central goal of mechanistic interpretability. However, scoring components one at a time misses context-dependent effects: a primary component can inhibit the activation of a backup, leading to issues with ranking components. Actual causality studies the structure of such interactions via witnesses: variables that provide contextual information to resolve interaction terms. However, estimation with witnesses typically requires combinatorial enumeration and is infeasible in practice. We introduce the witness-integrated set effect (WISE), a family of causal estimands that build on the witness mechanism while taking expectations over sets of causes and witnesses to remain computationally feasible. Building on this approach, we introduce JuntaLearner, a gradient-based circuit discovery method that learns to rank components by their causal impact across varying-sized sets of components and witnesses. Alongside faithfulness metrics, we introduce measures of necessity and task specificity, and the circuit recognition score (CRS) to summarize each metric across circuit sizes while emphasizing effects achieved by small circuits. Across tasks and models of increasing size, JuntaLearner achieves higher mean CRS compared to attribution baselines on all metrics. Since its cost does not grow with the number of candidate components, JuntaLearner scales to large models while accounting for set-level interactions and avoiding first-order approximations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 19 of its 30 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

19 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuits2 Jul
  2. 2026The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching25 Jun
  3. 2025Rethinking Circuit Completeness in Language Models: AND, OR, and ADDER Gates15 May
  4. 2023Attribution Patching Outperforms Automated Circuit Discovery16 Oct
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 14

  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  2. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  3. 2021Causal Abstractions of Neural Networkscited by 422
  4. 2013Direct and Indirect Effectscited by 280
  5. 2024How to use and interpret activation patchingcited by 214
  6. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156
  7. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  8. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  9. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 131
  10. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
  11. 2019Reparameterizable Subset Sampling via Continuous Relaxationscited by 70
  12. 2024Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networkscited by 23
2 more
  1. 2026A Computationally Feasible Framework for Causal Probabilistic Explanationcited by 1
  2. 2026Matryoshka attribution: Learning to attribute language model outputs to representations and weightscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Data-driven Circuit Discovery for Interpretability of Language Models
  2. 2026Finding Interpretable Prompt-Specific Circuits in Language Models
  3. 2026Circuit Condensation: Post-Training that Concentrates a Behavior's Causal Circuit
  4. 2025Position-aware Automatic Circuit Discovery
  5. 2025Attribution-Guided Pruning for Insight and Control: Circuit Discovery and Targeted Correction in Small-scale LLMs
  6. 2025Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagation
  7. 2026Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability
  8. 2026Architecture, Not Scale: Circuit Localization in Large Language Models