Interpreting vision transformers via residual replacement model
Not on the map. Why?
Abstract (arXiv)
How do vision transformers (ViTs) represent and process the world? This paper addresses this long-standing question through the first systematic analysis of 6.6K features across all layers, extracted via sparse autoencoders, and by introducing the residual replacement model, which replaces ViT computations with interpretable features in the residual stream. Our analysis reveals not only a feature evolution from low-level patterns to high-level semantics, but also how ViTs encode curves and spatial positions through specialized feature types. The residual replacement model scalably produces a faithful yet parsimonious circuit for human-scale interpretability by significantly simplifying the original computations. As a result, this framework enables intuitive understanding of ViT mechanisms. Finally, we demonstrate the utility of our framework in debiasing spurious correlations.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Sep 2025.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2025: 1; 2026: 3.
4 papers.
- 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
- 2025Sparse Autoencoders are Topic Modelscited by 3
- 2026Rethinking Prototype-based Similarity Learning for Few-Shot Object Detectioncited by 1
- 2026FailSAE: Towards Interpretable Failure Prediction for Vision-Language Models via Sparse Autoencoderscited by 0
- 2026FailSAE: Towards Interpretable Failure Prediction for Vision-Language Models via Sparse Autoencoders2 Sep 2026
- 2026Rethinking Prototype-based Similarity Learning for Few-Shot Object Detection22 Jun 2026
- 2026Interpretable Debiasing of Vision-Language Models for Social Fairness27 Feb 2026
- 2025Sparse Autoencoders are Topic Models20 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2026Inside the Visual Mind: Neuroscience-Motivated Concept Circuits for Interpreting and Steering Vision Transformers
- 2026Feature Evolution and Migration during Vision Transformer Training
- 2026Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision
- 2026ViT-5: Vision Transformers for The Mid-2020s
- 2023Interpretability-Aware Vision Transformer
- 2026ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers
- 2026Decoding the Functional Roles of Register and High-Norm Patch Tokens in Vision Transformers
- 2026From Edges to Depth: Probing the Spatial Hierarchy in Vision Transformers