# Locating and Editing Factual Associations in Mamba

arXiv: https://arxiv.org/abs/2404.03646 · HTML: https://mlsanity.com/paper/2404.03646/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Inducing Causal Structure for Interpretable Neural Networks](https://mlsanity.com/paper/2112.00826/) (2021)
- [The Hidden Attention of Mamba Models](https://mlsanity.com/paper/2403.01590/) (2024)
- [Is Mamba Capable of In-Context Learning?](https://mlsanity.com/paper/2402.03170/) (2024)

## Built on by (most cited first, 30 of 37)

- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Understanding Information Storage and Transfer in Multi-modal Large Language Models](https://mlsanity.com/paper/2406.04236/) (2024)
- [Visual Mamba: A Survey and New Outlooks](https://mlsanity.com/paper/2404.18861/) (2024)
- [Estimating Knowledge in Large Language Models Without Generating a Single Token](https://mlsanity.com/paper/2406.12673/) (2024)
- [In-Context Editing: Learning Knowledge from Self-Induced Distributions](https://mlsanity.com/paper/2406.11194/) (2024)
- [Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures](https://mlsanity.com/paper/2410.06672/) (2024)
- [Can Knowledge Editing Really Correct Hallucinations?](https://mlsanity.com/paper/2410.16251/) (2024)
- [Automatically Identifying Local and Global Circuits with Linear Computation Graphs](https://mlsanity.com/paper/2405.13868/) (2024)
- [The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis](https://mlsanity.com/paper/2408.01416/) (2024)
- [Can Editing LLMs Inject Harm?](https://mlsanity.com/paper/2407.20224/) (2024)
- [Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing](https://mlsanity.com/paper/2410.17194/) (2024)
- [From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries](https://mlsanity.com/paper/2406.12824/) (2024)
- [CoME: An Unlearning-based Approach to Conflict-free Model Editing](https://mlsanity.com/paper/2502.15826/) (2025)
- [Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks](https://mlsanity.com/paper/2408.04965/) (2024)
- [Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing](https://mlsanity.com/paper/2412.13341/) (2024)
- [Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamics](https://mlsanity.com/paper/2409.08530/) (2024)
- [Defining Knowledge: Bridging Epistemology and Large Language Models](https://mlsanity.com/paper/2410.02499/) (2024)
- [LLMs Process Lists With General Filter Heads](https://mlsanity.com/paper/2510.26784/) (2025)
- [Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling](https://mlsanity.com/paper/2510.26912/) (2025)
- [Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination](https://mlsanity.com/paper/2410.17477/) (2024)
- [Mamba Knockout for Unraveling Factual Information Flow](https://mlsanity.com/paper/2505.24244/) (2025)
- [Interpreting and Steering State-Space Models via Activation Subspace Bottlenecks](https://mlsanity.com/paper/2602.22719/) (2026)
- [Hidden State Poisoning Attacks against Mamba-based Language Models](https://mlsanity.com/paper/2601.01972/) (2026)
- [What Attention Recalls and Recurrence Controls in Hybrid Language Models](https://mlsanity.com/paper/2609.04434/) (2026)
- [Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models](https://mlsanity.com/paper/2510.12044/) (2025)
- [Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression](https://mlsanity.com/paper/2604.19089/) (2026)

## Abstract (arXiv)

We investigate the mechanisms of factual recall in the Mamba state space model. Our work is inspired by previous findings in autoregressive transformer language models suggesting that their knowledge recall is localized to particular modules at specific token locations; we therefore ask whether factual recall in Mamba can be similarly localized. To investigate this, we conduct four lines of experiments on Mamba. First, we apply causal tracing or interchange interventions to localize key components inside Mamba that are responsible for recalling facts, revealing that specific components within middle layers show strong causal effects at the last token of the subject, while the causal effect of intervening on later layers is most pronounced at the last token of the prompt, matching previous findings on autoregressive transformers. Second, we show that rank-one model editing methods can successfully insert facts at specific locations, again resembling findings on transformer LMs. Third, we examine the linearity of Mamba's representations of factual relations. Finally we adapt attention-knockout techniques to Mamba in order to dissect information flow during factual recall. We compare Mamba directly to a similar-sized autoregressive transformer LM and conclude that despite significant differences in architectural approach, when it comes to factual recall, the two architectures share many similarities.