mlsanityData to 7 Oct 2026

Interpreting and Steering State-Space Models via Activation Subspace Bottlenecks

arXiv 2602.22719 ↗First posted 26 Feb 2026cs.LGcited by 3

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

State-space models (SSMs) have emerged as an efficient strategy for building powerful language models, avoiding the quadratic complexity of computing attention in transformers. Despite their promise, the interpretability and steerability of modern SSMs remain relatively underexplored. We take a major step in this direction by identifying activation subspace bottlenecks in the Mamba family of SSM models using tools from mechanistic interpretability. We then introduce a test-time steering intervention that simply multiplies the activations of the identified bottlenecks by a scalar. Across 7 SSMs and 6 diverse benchmarks, this intervention improves performance by an average of 8.27%, without requiring any task-specific tuning. Finally, we validate that the identified bottlenecks are indeed hindering performance by modifying them to yield an architecture we call Stable-Mamba, which achieves long-context performance gains when retrained from scratch.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 61 of its 83 references that are in the map. First posted on arXiv 26 Feb 2026.

Builds on

61 of 83 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Characterizing Mamba's Selective Memory using Auto-Encoders17 Dec
  2. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data22 Sep
  3. 2025Mamba Knockout for Unraveling Factual Information Flow30 May
  4. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models14 Apr
  5. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mamba21 Feb
  6. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing31 Dec
  7. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures9 Oct
  8. 2024A Survey of Mamba2 Aug
  9. 2024MambaLRP: Explaining Selective State Space Sequence Models11 Jun
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  11. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation26 May
  12. 2024Locating and Editing Factual Associations in Mamba4 Apr
4 more
  1. 2024The Hidden Attention of Mamba Models3 Mar
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  3. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections24 Jun
  4. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 45

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  11. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  12. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
33 more
  1. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  2. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  3. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  4. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  5. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  6. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  7. 2024SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentationcited by 266
  8. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  9. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  10. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  11. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  12. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  13. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  14. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  15. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  16. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  17. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 73
  18. 2023Explaining black box text modules in natural language with language modelscited by 61
  19. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 56
  20. 2024Concept Bottleneck Large Language Modelscited by 41
  21. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  22. 2022Augmenting Interpretable Models with LLMs during Trainingcited by 36
  23. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  24. 2025Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decompositioncited by 25
  25. 2024Efficient Automated Circuit Discovery in Transformers using Contextual Decompositioncited by 23
  26. 2024DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Modelscited by 22
  27. 2024Bayesian Concept Bottleneck Models with LLM Priorscited by 16
  28. 2026Human-AI Co-design for Clinical Prediction Modelscited by 9
  29. 2023Centered Self-Attention Layerscited by 7
  30. 2024Investigating the Indirect Object Identification circuit in Mambacited by 5
  31. 2025In-Context Algebracited by 4
  32. 2024Interpretable Next-token Prediction via the Generalized Induction Headcited by 2
  33. 2025FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependenciescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026WriteSAE: Sparse Autoencoders for Recurrent Statecited by 1
  2. 2026A Circuit, Not The Circuit: Non-Unique Causal Localisation of the Mamba-2 State Sinkcited by 0
  3. 2026Register Tokens for Bounded-State Reasoning in Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024An Empirical Study of Mamba-based Language Models
  2. 2026Towards Understanding What State Space Models Learn About Code
  3. 2024The Expressive Capacity of State Space Models: A Formal Language Perspective
  4. 2026Benchmarking the Computational and Representational Efficiency of State Space Models against Transformers on Long-Context Dyadic Sessions
  5. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
  6. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Models
  7. 2024Sparse Mamba: Introducing Controllability, Observability, And Stability To Structural State Space Models
  8. 2025Controllability Analysis of State Space-based Language Model