mlsanityData to 7 Oct 2026

Investigating the Indirect Object Identification circuit in Mamba

arXiv 2407.14008 ↗First posted 19 Jul 2024cs.LGcited by 5

On the map: State-space sequence models (S4, Mamba) → Vision Mamba backbones

Abstract (arXiv)

How well will current interpretability techniques generalize to future models? A relevant case study is Mamba, a recent recurrent architecture with scaling comparable to Transformers. We adapt pre-Mamba techniques to Mamba and partially reverse-engineer the circuit responsible for the Indirect Object Identification (IOI) task. Our techniques provide evidence that 1) Layer 39 is a key bottleneck, 2) Convolutions in layer 39 shift names one position forward, and 3) The name entities are stored linearly in Layer 39's SSM. Finally, we adapt an automatic circuit discovery tool, positional Edge Attribution Patching, to identify a Mamba IOI circuit. Our contributions provide initial evidence that circuit-based mechanistic interpretability tools work well for the Mamba architecture.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 14 of its 35 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

14 of 35 references.

References · 14

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  8. 2019Are Sixteen Heads Really Better than One?cited by 803
  9. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  10. 2021Causal Abstractions of Neural Networkscited by 401
  11. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  12. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
2 more
  1. 2021Low-Complexity Probing via Finding Subnetworkscited by 63
  2. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Interpreting and Steering State-Space Models via Activation Subspace Bottleneckscited by 3
  2. 2026Hidden State Poisoning Attacks against Mamba-based Language Modelscited by 2
  3. 2026Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoningcited by 1
  4. 2026A Circuit, Not The Circuit: Non-Unique Causal Localisation of the Mamba-2 State Sinkcited by 0
  5. 2026The Key Handoff: Retrieval in Hybrid Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
  2. 2024MambaOut: Do We Really Need Mamba for Vision?
  3. 2024The Hidden Attention of Mamba Models
  4. 2024Mamba-R: Vision Mamba ALSO Needs Registers
  5. 2025MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modeling
  6. 2026SF-Mamba: Rethinking State Space Model for Vision
  7. 2025RNN as Linear Transformer: A Closer Investigation into Representational Potentials of Visual Mamba Models
  8. 2025A Survey on Mamba Architecture for Vision Applications