mlsanityData to 14 Oct 2026

WriteSAE: Sparse Autoencoders for Recurrent State

arXiv 2605.12770 ↗First posted 12 May 2026cs.LG cs.AI cs.CLcited by 1

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

also in 1 more
Abstract (arXiv)

We introduce WriteSAE, a sparse autoencoder for the matrix updates written into recurrent language-model state. In Gated DeltaNet, Mamba-2, and RWKV-7, each token writes a matrix-shaped update to a recurrent cache; a residual-stream SAE has vector-shaped atoms and cannot replace that update directly. WriteSAE learns rank-1 matrix atoms with the same shape as the model's own write. This lets us test a direct replacement: at positions where the SAE activates an atom, we remove the model's write, insert the atom scaled by its SAE activation, and continue the forward pass. The atom gives a closer final token distribution than deleting the write on 92.4% of evaluated positions; averaged per atom, the rate is 89.8%. For Gated DeltaNet, a formula using the forget gate, read query, and output embedding predicts the resulting logit change with R2 = 0.98. The same replacement test transfers to Mamba-2-370M at 88.1%. In generation, the formula chooses a write direction; writing it into three consecutive cache positions at 3× the norm of the model's write makes tokens initially ranked 100--1000 by the unmodified model appear in 100% of continuations, up from 33.3%. To our knowledge this is the first cache-level steering intervention reported in a state-space or hybrid recurrent layer.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 59 of its 70 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

59 of 70 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026Mamba-3: Improved Sequence Modeling using State Space Principles16 Mar
  2. 2026Interpreting and Steering State-Space Models via Activation Subspace Bottlenecks26 Feb
  3. 2025Characterizing Mamba's Selective Memory using Auto-Encoders17 Dec
  4. 2025Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View18 Jun
  5. 2025Mechanistic evaluation of Transformers and state space models21 May
  6. 2025LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models21 Feb
  7. 2024Parameter-Efficient Fine-Tuning of State Space Models11 Oct
  8. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures9 Oct
  9. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden States5 Jul
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  11. 2024Does Transformer Interpretability Transfer to RNNs?9 Apr
  12. 2024Locating and Editing Factual Associations in Mamba4 Apr
1 more
  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec

Other references · 46

  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2022Mass-Editing Memory in a Transformercited by 1,001
  4. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  5. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  6. 2024Scaling and evaluating sparse autoencoderscited by 553
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  9. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  10. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  11. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  12. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
34 more
  1. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  2. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  3. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  4. 2013k-Sparse Autoencoderscited by 215
  5. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  6. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  7. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  8. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  9. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 179
  10. 2016Using Fast Weights to Attend to the Recent Pastcited by 170
  11. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 143
  12. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  13. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 124
  14. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 120
  15. 2025Sparse Autoencoders Trained on the Same Data Learn Different Featurescited by 93
  16. 2022Polysemanticity and Capacity in Neural Networkscited by 77
  17. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 75
  18. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 47
  19. 2024Mamba-R: Vision Mamba ALSO Needs Registerscited by 41
  20. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 27
  21. 2025Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decompositioncited by 27
  22. 2025Transcoders Beat Sparse Autoencoders for Interpretabilitycited by 25
  23. 2025Dense SAE Latents Are Features, Not Bugscited by 17
  24. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 15
  25. 2026Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMscited by 12
  26. 2025Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activationscited by 10
  27. 2025Towards Understanding the Nature of Attention with Low-Rank Sparse Decompositioncited by 9
  28. 2025Revisiting associative recall in modern recurrent modelscited by 8
  29. 2026On State Reduction in Linear Attentioncited by 6
  30. 2025Finding Manifolds With Bilinear Autoencoderscited by 5
  31. 2026PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decodingcited by 3
  32. 2015Efficient Sum of Outer Products Dictionary Learning (SOUP-DIL) - The ℓ0 Methodcited by 1
  33. 2025Evaluating Sparse Autoencoders: From Shallow Design to Matching Pursuitcited by 1
  34. 2026Behavioral Steering in a 35B MoE Language Model via SAE-Decoded Probe Vectors: One Agency Axis, Not Five Traitscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replaycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Low-Rank Adapting Models for Sparse Autoencoders
  2. 2025Towards Atoms of Large Language Models
  3. 2026Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations
  4. 2026Diminishing Returns of Data in Sparse Autoencoders: Functional Behavior does not Guarantee Feature Reproducibility
  5. 2026MARS: Enabling Autoregressive Models Multi-Token Generation
  6. 2024Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders
  7. 2026When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs
  8. 2026Compressed-Sensing-Guided, Inference-Aware Structured Reduction for Large Language Models