mlsanityData to 14 Oct 2026

The Independence Prior of SAEs Fragments Visual Concepts

arXiv 2610.04112 ↗First posted 2 Oct 2026cs.AIcited by 0

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Sparse Autoencoders (SAEs) decompose model activations into sparse combinations of interpretable dictionary atoms. Although SAEs are grounded in the Linear Representation Hypothesis (LRH), their objective smuggles in an additional prior: concepts across patches are treated as independent, an assumption clearly violated by natural images and by the activations they induce. We therefore specialize LRH to vision through the Markov-Field Linear Representation Hypothesis (MFLRH), which adds the missing spatial dependencies to the LRH assumptions. We thus propose Spatial-SAE as an amortized MAP estimator under the MFLRH. Spatial-SAE consistently outperforms standard SAEs in concept recovery and interpretability. Across four variants, it achieves a 96% average win rate on synthetic concept recovery and improves interpretability on DINOv2 activations, at a reconstruction cost concentrated in high spatial frequencies.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 26 of its 46 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

26 of 46 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Do Sparse Autoencoders Capture Concept Manifolds?30 Apr
  2. 2026SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data16 Feb
  3. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit3 Jun
  4. 2025Feature Hedging: Correlated Features Break Narrow Sparse Autoencoders16 May
  5. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry3 Mar
  6. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models18 Feb
  7. 2024Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders20 Nov
  8. 2024Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations with MDL-SAEs15 Oct
  9. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 29 Aug

Other references · 17

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  5. 2023Vision Transformers Need Registerscited by 871
  6. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  7. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  8. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 160
  9. 2022Vision Transformers provably learn spatial structurecited by 93
  10. 2024On the Origins of Linear Representations in Large Language Modelscited by 79
  11. 2025Into the Rabbit Hull: From Task-Relevant Concepts in DINO to Minkowski Geometrycited by 34
  12. 2023Amortized Variational Inference: When and Why?cited by 29
5 more
  1. 2015Submodular Functions: from Discrete to Continous Domainscited by 25
  2. 2025Interpretable and Testable Vision Features via Sparse Autoencoderscited by 25
  3. 2025Priors in Time: Missing Inductive Biases for Language Model Interpretabilitycited by 25
  4. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 21
  5. 2026The Rate-Distortion-Polysemanticity Tradeoff in SAEscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?
  2. 2025Probing the Representational Power of Sparse Autoencoders in Vision Models
  3. 2025Mechanistic Interpretability with Sparse Autoencoder Neural Operators
  4. 2026When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities
  5. 2025Concept-SAE: A Controllable and Invertible Concept Interface for Sparse Autoencoders
  6. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?
  7. 2025Interpretable and Steerable Concept Bottleneck Sparse Autoencoders
  8. 2024Interpreting Attention Layer Outputs with Sparse Autoencoders