A Circuit, Not The Circuit: Non-Unique Causal Localisation of the Mamba-2 State Sink
On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language
also in 1 more
Abstract (arXiv)
Mechanistic interpretability routinely reads a probe and labels its top-activating units as the circuit executing the computation. We test the move in Mamba, on the state sink: the selective state-space analogue of the Transformer attention sink, where the Delta-gate fires disproportionately on boundary tokens such as BOS and newline. At Mamba-1 channel granularity the probe's units carry the causal effect. At Mamba-2 head granularity the label-to-locus link breaks in three ways. The causal set is not unique: a near-disjoint set of top-activation heads, sharing almost none of the specialists, matches or exceeds their ablation effect. Representation does not track function: dual heads, a quarter to over a third of all heads, respond alike to BOS and newline yet do not reproduce the specialists' causal profile. And the effect dissociates by intervention surface: perturbing the input-dependent Delta the probe reads moves the loss by at most 0.16 nats, while suppressing the same heads' output moves it by 1.5 to 7.3 nats. The heads still matter behaviourally, dropping needle-retrieval success on a RULER-style probe from 30/30 to 0/30 at 1024 context on both flagship checkpoints. A representational signature in a selective state-space model thus marks a behaviourally important head set without pinning down the circuit: a single probe recovers a circuit, not the circuit.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 44 of its 50 references that are in the map. First posted on arXiv 30 May 2026.
Builds on
44 of 50 references.
Closest ancestors · 12
Same direction, newest first
- 2026Interpreting and Steering State-Space Models via Activation Subspace Bottlenecks26 Feb
- 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement22 Apr
- 2025LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models21 Feb
- 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Models17 Oct
- 2024Falcon Mamba: The First Competitive Attention-free 7B Language Model7 Oct
- 2024An Empirical Study of Mamba-based Language Models12 Jun
- 2024MambaLRP: Explaining Selective State Space Sequence Models11 Jun
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
- 2024Locating and Editing Factual Associations in Mamba4 Apr
- 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
- 2024The Hidden Attention of Mamba Models3 Mar
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
Other references · 32
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
- 2022In-context Learning and Induction Headscited by 964
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2023Vision Transformers Need Registerscited by 871
20 more
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
- 2021Causal Abstractions of Neural Networkscited by 422
- 2024Massive Activations in Large Language Modelscited by 294
- 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
- 2023LEACE: Perfect linear concept erasure in closed formcited by 267
- 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
- 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
- 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
- 2024How to use and interpret activation patchingcited by 214
- 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
- 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
- 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
- 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
- 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
- 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 64
- 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 56
- 2024StableMask: Refining Causal Masking in Decoder-only Transformercited by 20
- 2024Investigating the Indirect Object Identification circuit in Mambacited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Finding Interpretable Prompt-Specific Circuits in Language Models
- 2025Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers
- 2026Architecture, Not Scale: Circuit Localization in Large Language Models
- 2026Single-Position Intervention Fails: Distributed Output Templates Drive In-Context Learning
- 2026Pattern Selectivity is Not Task-Causal Structure: A Cross-Architecture Mechanistic Study of Composed-Task Circuits in 1B-Class Language Models
- 2026Lagged Coupling: Internal Representations Become Readable Before They Become Causal
- 2026Same Outcome, Different Readout: What Does a Steerable Valence Direction in LLMs Represent?
- 2026Predictive Memory Localization: Forecasting Selective Intervention Paths from Internal Signals