# Analyzing Vision Transformers for Image Classification in Class Embedding Space

arXiv: https://arxiv.org/abs/2310.18969 · HTML: https://mlsanity.com/paper/2310.18969/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 22 references that are in the map.
- First posted on arXiv 29 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)
- [Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?](https://mlsanity.com/paper/2005.00719/) (2020)
- [Analyzing Transformers in Embedding Space](https://mlsanity.com/paper/2209.02535/) (2022)
- [On Identifiability in Transformers](https://mlsanity.com/paper/1908.04211/) (2019)
- [Large-scale Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2106.03149/) (2021)
- [Jump to Conclusions: Short-Cutting Transformers With Linear Transformations](https://mlsanity.com/paper/2303.09435/) (2023)
- [What do Vision Transformers Learn? A Visual Exploration](https://mlsanity.com/paper/2212.06727/) (2022)
- [On Improving Adversarial Transferability of Vision Transformers](https://mlsanity.com/paper/2106.04169/) (2021)
- [Overinterpretation reveals image classification model pathologies](https://mlsanity.com/paper/2003.08907/) (2020)

## Built on by (most cited first, 26 of 26)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Towards Interpreting Visual Information Processing in Vision-Language Models](https://mlsanity.com/paper/2410.07149/) (2024)
- [Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?](https://mlsanity.com/paper/2502.20914/) (2025)
- [Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts](https://mlsanity.com/paper/2505.17127/) (2025)
- [What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation](https://mlsanity.com/paper/2406.16320/) (2024)
- [Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features](https://mlsanity.com/paper/2410.03558/) (2024)
- [Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Video](https://mlsanity.com/paper/2504.19475/) (2025)
- [Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP](https://mlsanity.com/paper/2406.01583/) (2024)
- [Block-Recurrent Dynamics in Vision Transformers](https://mlsanity.com/paper/2512.19941/) (2025)
- [Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects](https://mlsanity.com/paper/2406.15955/) (2024)
- [Looking Beyond The Top-1: Transformers Determine Top Tokens In Order](https://mlsanity.com/paper/2410.20210/) (2024)
- [Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neuroscience](https://mlsanity.com/paper/2406.01352/) (2024)
- [LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate](https://mlsanity.com/paper/2405.13985/) (2024)
- [Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokens](https://mlsanity.com/paper/2412.04680/) (2024)
- [Signatures of human-like processing in Transformer forward passes](https://mlsanity.com/paper/2504.14107/) (2025)
- [LLMs Can Compensate for Deficiencies in Visual Representations](https://mlsanity.com/paper/2506.05439/) (2025)
- [Self-Distillation of Hidden Layers for Self-Supervised Representation Learning](https://mlsanity.com/paper/2603.15553/) (2026)
- [Convolutional Neural Networks and Vision Transformers for Fashion MNIST Classification: A Literature Review](https://mlsanity.com/paper/2406.03478/) (2024)
- [CoRLD: Contrastive Representation Learning Of Deformable Shapes In Images](https://mlsanity.com/paper/2503.17162/) (2025)
- [Communication Efficient Split Learning of ViTs with Attention-based Double Compression](https://mlsanity.com/paper/2509.15058/) (2025)
- [FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks](https://mlsanity.com/paper/2408.02123/) (2024)
- [ECG-Soup: Harnessing Multi-Layer Synergy for ECG Foundation Models](https://mlsanity.com/paper/2509.00102/) (2025)
- [Unveiling Transformer Perception by Exploring Input Manifolds](https://mlsanity.com/paper/2410.06019/) (2024)
- [How Can Multimodal Remote Sensing Datasets Transform Classification via SpatialNet-ViT?](https://mlsanity.com/paper/2506.22501/) (2025)
- [CircuitProbe: Tracing Visual Temporal Evidence Flow in Video Language Models](https://mlsanity.com/paper/2507.19420/) (2025)

## Abstract (arXiv)

Despite the growing use of transformer models in computer vision, a mechanistic understanding of these networks is still needed. This work introduces a method to reverse-engineer Vision Transformers trained to solve image classification tasks. Inspired by previous research in NLP, we demonstrate how the inner representations at any level of the hierarchy can be projected onto the learned class embedding space to uncover how these networks build categorical representations for their predictions. We use our framework to show how image tokens develop class-specific representations that depend on attention mechanisms and contextual information, and give insights on how self-attention and MLP layers differentially contribute to this categorical composition. We additionally demonstrate that this method (1) can be used to determine the parts of an image that would be important for detecting the class of interest, and (2) exhibits significant advantages over traditional linear probing approaches. Taken together, our results position our proposed framework as a powerful tool for mechanistic interpretability and explainability research.