Finding Usable Weight Mechanisms with Tiled SVD
On the map: Mechanistic interpretability → Circuits and mechanistic analysis
Abstract (arXiv)
The dominant approach to mechanistic interpretability trains proxy dictionaries such as sparse autoencoders and labels features from max-activating text. The best such atlases identify con- cepts, but that identity lives in the learned dictionary rather than in the network weights them- selves. We propose extracting mechanism mounts directly from linear sites by column-tiled SVD: each mount is a triple (v,u,σ) read as trigger, write, and strength. Identity is the weight rule. We evaluate mounts with a pre-registered suite judged on full-write energy lift rather than tile-local lift. On Gemma-2-2B with WikiText-2 (16,384-token subsample), all seven linear maps are scored: residual writes (mlp.down, attn.o) receive full A/B/C with steer after post-sublayer RMSNorm and pass 52/52 site-layers; other maps receive A/B only (mlp.gate/attn.q/attn.k/effective mlp.up/attn.v 26/26 each). Aggregate: 182/182 GO. We release library code, the corpus builder, the experiment entrypoint, and unit tests.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 2 of its 5 references that are in the map. First posted on arXiv 7 Aug 2026.
Builds on
2 of 5 references.
Closest ancestors · 1
Same direction, newest first
Other references · 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural Constraints
- 2025Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition
- 2026Shared Semantics, Divergent Mechanisms: Unsupervised Feature Discovery by Aligning Semantics and Mechanisms
- 2026Representation as a Bottleneck for Mechanistic Interpretability: The Manifestation Unit Protocol
- 2026Individual Parameters in Weight-Sparse Transformers Appear Interpretable
- 2024Bilinear MLPs enable weight-based mechanistic interpretability
- 2026Weight Oracles: Reading Neural Network Weights with Language Models
- 2026Certified Mechanistic Interpretability: Lifting Single-Input Findings to Bounded Neighbourhoods