mlsanityData to 14 Oct 2026

A Deeper Analysis of Block-Sparse Featurizers

arXiv 2608.27515 ↗First posted 27 Aug 2026cs.LG cs.CVcited by 0

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

The recently introduced block-sparse featurizer (BSF; Fel et al., 2026) is similar to a sparse autoencoder (SAE), but its atomic unit is a small subspace (a block of directions) rather than a single direction. It is designed for features that live on low-dimensional manifolds, which are especially frequent in vision. This work studies the BSF's strengths and weaknesses, finding how it still somewhat suffers from classic SAE failure modes, like feature splitting and composition. We propose several architectural changes to the BSF, including a Tournament Top-K selection rule that significantly reduces feature splitting, and we also extend the block paradigm to the crosscoder.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 10 of its 14 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

10 of 14 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Do Sparse Autoencoders Capture Concept Manifolds?30 Apr
  2. 2025OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features26 Sep
  3. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit3 Jun
  4. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoders21 Mar
  5. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models18 Feb

Other references · 5

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2025DINOv3cited by 1,414
  3. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  4. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 160
  5. 2026Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifoldscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models ↗
  2. 2025Sparse Autoencoders, Again?
  3. 2026Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders
  4. 2025Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features Alone
  5. 2026Aligned Training: A Parameter-Free Method to Improve Feature Quality and Stability of Sparse Autoencoders (SAE)
  6. 2026LookThere! Sparse Vision by Reinforced Selection
  7. 2025Steering CLIP's vision transformer with sparse autoencoders
  8. 2025Analysis of Variational Sparse Autoencoders