# Beyond Semantics: Disentangling Information Scope in Sparse Autoencoders for CLIP

arXiv: https://arxiv.org/abs/2604.05724 · HTML: https://mlsanity.com/paper/2604.05724/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Interpreting vision and multimodal models, one of 13 directions in Mechanistic interpretability.
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Placed in

- Mechanistic interpretability / [Interpreting vision and multimodal models](https://mlsanity.com/mechinterp/interpreting-vision-and-multimodal-models/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery](https://mlsanity.com/paper/2407.14499/) (2024)
- [Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models](https://mlsanity.com/paper/2504.02821/) (2025)
- [Sparse autoencoders reveal selective remapping of visual concepts during adaptation](https://mlsanity.com/paper/2412.05276/) (2024)
- [Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment](https://mlsanity.com/paper/2502.03714/) (2025)
- [Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry](https://mlsanity.com/paper/2503.01822/) (2025)
- [Interpreting CLIP with Hierarchical Sparse Autoencoders](https://mlsanity.com/paper/2502.20578/) (2025)
- [Vision Transformers Don't Need Trained Registers](https://mlsanity.com/paper/2506.08010/) (2025)
- [Large Multi-modal Models Can Interpret Features in Large Multi-modal Models](https://mlsanity.com/paper/2411.14982/) (2024)
- [Probing the Representational Power of Sparse Autoencoders in Vision Models](https://mlsanity.com/paper/2508.11277/) (2025)
- [Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval](https://mlsanity.com/paper/2506.00041/) (2025)
- [Causal Interpretation of Sparse Autoencoder Features in Vision](https://mlsanity.com/paper/2509.00749/) (2025)
- [Pruning the Paradox: How CLIP's Most Informative Heads Enhance Performance While Amplifying Bias](https://mlsanity.com/paper/2503.11103/) (2025)

## Built on by (most cited first, 1 of 1)

- [Geometric Similarity in VLM Low-Level Vision Representations](https://mlsanity.com/paper/2610.00848/) (2026)

## Abstract (arXiv)

Sparse Autoencoders (SAEs) have emerged as a powerful tool for interpreting the internal representations of CLIP vision encoders, yet existing analyses largely focus on the semantic meaning of individual features. We introduce information scope as a complementary dimension of interpretability that characterizes how broadly an SAE feature aggregates visual evidence, ranging from localized, patch-specific cues to global, image-level signals. We observe that some SAE features respond consistently across spatial perturbations, while others shift unpredictably with minor input changes, indicating a fundamental distinction in their underlying scope. To quantify this, we propose the Contextual Dependency Score (CDS), which separates positionally stable local scope features from positionally variant global scope features. Our experiments show that features of different information scopes exert systematically different influences on CLIP's predictions and confidence. These findings establish information scope as a critical new axis for understanding CLIP representations and provide a deeper diagnostic view of SAE-derived features.