# Residual Stream Analysis with Multi-Layer SAEs

arXiv: https://arxiv.org/abs/2409.04185 · HTML: https://mlsanity.com/paper/2409.04185/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 36 of its 56 references that are in the map.
- First posted on arXiv 6 Sep 2024.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Extracting Latent Steering Vectors from Pretrained Language Models](https://mlsanity.com/paper/2205.05124/) (2022)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors](https://mlsanity.com/paper/2103.15949/) (2021)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations](https://mlsanity.com/paper/2402.17700/) (2024)
- [Residual Connections Encourage Iterative Inference](https://mlsanity.com/paper/1710.04773/) (2017)
- [Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning](https://mlsanity.com/paper/2405.12241/) (2024)
- [Interpreting Attention Layer Outputs with Sparse Autoencoders](https://mlsanity.com/paper/2406.17759/) (2024)
- [Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small](https://mlsanity.com/paper/2409.04478/) (2024)

## Built on by (most cited first, 19 of 19)

- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Beyond Linear Steering: Unified Multi-Attribute Control for Language Models](https://mlsanity.com/paper/2505.24535/) (2025)
- [Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders](https://mlsanity.com/paper/2505.16004/) (2025)
- [Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups](https://mlsanity.com/paper/2410.21508/) (2024)
- [Transformer Dynamics: A neuroscientific approach to interpretability of large language models](https://mlsanity.com/paper/2502.12131/) (2025)
- [Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations](https://mlsanity.com/paper/2502.18147/) (2025)
- [Learning to Skip the Middle Layers of Transformers](https://mlsanity.com/paper/2506.21103/) (2025)
- [UniFusion: Vision-Language Model as Unified Encoder in Image Generation](https://mlsanity.com/paper/2510.12789/) (2025)
- [Understanding Multi-View Transformers](https://mlsanity.com/paper/2510.24907/) (2025)
- [Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology](https://mlsanity.com/paper/2605.14258/) (2026)
- [go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices](https://mlsanity.com/paper/2604.02309/) (2026)
- [Visual Exploration of Feature Relationships in Sparse Autoencoders with Curated Concepts](https://mlsanity.com/paper/2511.06048/) (2025)
- [SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models](https://mlsanity.com/paper/2606.32022/) (2026)
- [MonoLoss: A Training Objective for Interpretable Monosemantic Representations](https://mlsanity.com/paper/2602.12403/) (2026)
- [Activation Transport Operators](https://mlsanity.com/paper/2508.17540/) (2025)
- [Rethinking Intrinsic Dimension Estimation in Neural Representations](https://mlsanity.com/paper/2604.20276/) (2026)
- [fmxcoders: Factorized Masked Crosscoders for Cross-Layer Feature Discovery](https://mlsanity.com/paper/2605.09438/) (2026)
- [ScopeSAE: Model-Scope Feature Discovery with Interpretable Layer Selection](https://mlsanity.com/paper/2610.04905/) (2026)

## Abstract (arXiv)

Sparse autoencoders (SAEs) are a promising approach to interpreting the internal representations of transformer language models. However, SAEs are usually trained separately on each transformer layer, making it difficult to use them to study how information flows across layers. To solve this problem, we introduce the multi-layer SAE (MLSAE): a single SAE trained on the residual stream activation vectors from every transformer layer. Given that the residual stream is understood to preserve information across layers, we expected MLSAE latents to 'switch on' at a token position and remain active at later layers. Interestingly, we find that individual latents are often active at a single layer for a given token or prompt, but the layer at which an individual latent is active may differ for different tokens or prompts. We quantify these phenomena by defining a distribution over layers and considering its variance. We find that the variance of the distributions of latent activations over layers is about two orders of magnitude greater when aggregating over tokens compared with a single token. For larger underlying models, the degree to which latents are active at multiple layers increases, which is consistent with the fact that the residual stream activation vectors at adjacent layers become more similar. Finally, we relax the assumption that the residual stream basis is the same at every layer by applying pre-trained tuned-lens transformations, but our findings remain qualitatively similar. Our results represent a new approach to understanding how representations change as they flow through transformers. We release our code to train and analyze MLSAEs at https://github.com/tim-lawson/mlsae.