mlsanityData to 14 Oct 2026

Finding Usable Weight Mechanisms with Tiled SVD

arXiv 2608.06969 ↗First posted 7 Aug 2026cs.AIcited by 0

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

The dominant approach to mechanistic interpretability trains proxy dictionaries such as sparse autoencoders and labels features from max-activating text. The best such atlases identify con- cepts, but that identity lives in the learned dictionary rather than in the network weights them- selves. We propose extracting mechanism mounts directly from linear sites by column-tiled SVD: each mount is a triple (v,u,σ) read as trigger, write, and strength. Identity is the weight rule. We evaluate mounts with a pre-registered suite judged on full-write energy lift rather than tile-local lift. On Gemma-2-2B with WikiText-2 (16,384-token subsample), all seven linear maps are scored: residual writes (mlp.down, attn.o) receive full A/B/C with steer after post-sublayer RMSNorm and pass 52/52 site-layers; other maps receive A/B only (mlp.gate/attn.q/attn.k/effective mlp.up/attn.v 26/26 each). Aggregate: 182/182 GO. We release library code, the corpus builder, the experiment entrypoint, and unit tests.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 2 of its 5 references that are in the map. First posted on arXiv 7 Aug 2026.

Builds on

2 of 5 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits25 Nov

Other references · 1

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural Constraints
  2. 2025Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition
  3. 2026Shared Semantics, Divergent Mechanisms: Unsupervised Feature Discovery by Aligning Semantics and Mechanisms
  4. 2026Representation as a Bottleneck for Mechanistic Interpretability: The Manifestation Unit Protocol
  5. 2026Individual Parameters in Weight-Sparse Transformers Appear Interpretable
  6. 2024Bilinear MLPs enable weight-based mechanistic interpretability
  7. 2026Weight Oracles: Reading Neural Network Weights with Language Models
  8. 2026Certified Mechanistic Interpretability: Lifting Single-Input Findings to Bounded Neighbourhoods