# Analyze Feature Flow to Enhance Interpretation and Steering in Language Models

arXiv: https://arxiv.org/abs/2502.03032 · HTML: https://mlsanity.com/paper/2502.03032/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 19 of 19)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024)
- [BatchTopK Sparse Autoencoders](https://mlsanity.com/paper/2412.06410/) (2024)
- [Improving Steering Vectors by Targeting Sparse Autoencoder Features](https://mlsanity.com/paper/2411.02193/) (2024)
- [Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures](https://mlsanity.com/paper/2410.06672/) (2024)
- [Automatically Identifying Local and Global Circuits with Linear Computation Graphs](https://mlsanity.com/paper/2405.13868/) (2024)
- [Mechanistic Permutability: Match Features Across Layers](https://mlsanity.com/paper/2410.07656/) (2024)
- [Evolution of SAE Features Across Layers in LLMs](https://mlsanity.com/paper/2410.08869/) (2024)

## Built on by (most cited first, 13 of 13)

- [Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality](https://mlsanity.com/paper/2503.24277/) (2025)
- [Activation-Space Personality Steering: Hybrid Layer Selection for Stable Trait Control in LLMs](https://mlsanity.com/paper/2511.03738/) (2025)
- [FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies](https://mlsanity.com/paper/2506.17673/) (2025)
- [Cross-Layer Discrete Concept Discovery for Interpreting Language Models](https://mlsanity.com/paper/2506.20040/) (2025)
- [Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression](https://mlsanity.com/paper/2605.28567/) (2026)
- [Scalable Circuit Learning for Interpreting Large Language Models](https://mlsanity.com/paper/2606.16939/) (2026)
- [DePass: Unified Feature Attributing by Simple Decomposed Forward Pass](https://mlsanity.com/paper/2510.18462/) (2025)
- [Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders](https://mlsanity.com/paper/2606.12138/) (2026)
- [CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits](https://mlsanity.com/paper/2608.05732/) (2026)
- [Domain-Filtered Knowledge Graphs from Sparse Autoencoder Features](https://mlsanity.com/paper/2604.23829/) (2026)
- [Rational Sparse Autoencoder](https://mlsanity.com/paper/2606.14990/) (2026)
- [Where Decoder Cosine Similarity Fails for SAE Feature Flow Discovery](https://mlsanity.com/paper/2609.12591/) (2026)
- [Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space](https://mlsanity.com/paper/2609.37731/) (2026)

## Abstract (arXiv)

We introduce a new approach to systematically map features discovered by sparse autoencoder across consecutive layers of large language models, extending earlier work that examined inter-layer feature links. By using a data-free cosine similarity technique, we trace how specific features persist, transform, or first appear at each stage. This method yields granular flow graphs of feature evolution, enabling fine-grained interpretability and mechanistic insights into model computations. Crucially, we demonstrate how these cross-layer feature maps facilitate direct steering of model behavior by amplifying or suppressing chosen features, achieving targeted thematic control in text generation. Together, our findings highlight the utility of a causal, cross-layer interpretability framework that not only clarifies how features develop through forward passes but also provides new means for transparent manipulation of large language models.