# From Colors to Classes: Emergence of Concepts in Vision Transformers

arXiv: https://arxiv.org/abs/2503.24071 · HTML: https://mlsanity.com/paper/2503.24071/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [CRAFT: Concept Recursive Activation FacTorization for Explainability](https://mlsanity.com/paper/2211.10154/) (2022)
- [CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks](https://mlsanity.com/paper/2204.10965/) (2022)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [Are Convolutional Neural Networks or Transformers more like human vision?](https://mlsanity.com/paper/2105.07197/) (2021)
- [Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees](https://mlsanity.com/paper/2301.11911/) (2023)
- [Labeling Neural Representations with Inverse Recognition](https://mlsanity.com/paper/2311.13594/) (2023)
- [CoSy: Evaluating Textual Explanations of Neurons](https://mlsanity.com/paper/2405.20331/) (2024)
- [Understanding Visual Feature Reliance through the Lens of Complexity](https://mlsanity.com/paper/2407.06076/) (2024)
- [Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers](https://mlsanity.com/paper/2412.06639/) (2024)

## Built on by (most cited first, 23 of 23)

- [From What to How: Attributing CLIP's Latent Components Reveals Unexpected Semantic Reliance](https://mlsanity.com/paper/2505.20229/) (2025)
- [Unlocking Pretrained Vision Transformers for Time Series Classification](https://mlsanity.com/paper/2506.08641/) (2025)
- [Granular Concept Circuits: Toward a Fine-Grained Circuit Discovery for Concept Representations](https://mlsanity.com/paper/2508.01728/) (2025)
- [Large Vision Models Can Solve Mental Rotation Problems](https://mlsanity.com/paper/2509.15271/) (2025)
- [Attentive multilayer fusion for vision transformers](https://mlsanity.com/paper/2601.09322/) (2026)
- [FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens](https://mlsanity.com/paper/2506.03096/) (2025)
- [Superposition Without Interference? Towards Isolated Interventions via Almost Orthogonal Features in Language Models](https://mlsanity.com/paper/2602.04718/) (2026)
- [Image Reconstruction as a Tool for Feature Analysis](https://mlsanity.com/paper/2506.07803/) (2025)
- [Neutralizing Token Aggregation via Information Augmentation for Efficient Test-Time Adaptation](https://mlsanity.com/paper/2508.03388/) (2025)
- [MIMIC: Multimodal Inversion for Model Interpretation and Conceptualization](https://mlsanity.com/paper/2508.07833/) (2025)
- [From independent patches to coordinated attention: Controlling information flow in vision transformers](https://mlsanity.com/paper/2602.04784/) (2026)
- [Seeing Through Circuits: Faithful Mechanistic Interpretability for Vision Transformers](https://mlsanity.com/paper/2604.14477/) (2026)
- [Rethinking Prototype-based Similarity Learning for Few-Shot Object Detection](https://mlsanity.com/paper/2606.23069/) (2026)
- [Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models](https://mlsanity.com/paper/2608.06901/) (2026)
- [Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models](https://mlsanity.com/paper/2509.21102/) (2025)
- [A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networks](https://mlsanity.com/paper/2510.24342/) (2025)
- [Depth-Wise Representation Development Under Blockwise Self-Supervised Learning for Video Vision Transformers](https://mlsanity.com/paper/2601.09040/) (2026)
- [Do Pathology Foundation Models Encode Disease Progression? A Pseudotime Analysis of Visual Representations](https://mlsanity.com/paper/2601.21334/) (2026)
- [Understanding Task Aggregation for Generalizable Ultrasound Foundation Models](https://mlsanity.com/paper/2603.18123/) (2026)
- [Separate Aggregation of Split Network for Personalized Federated Learning](https://mlsanity.com/paper/2605.26571/) (2026)
- [What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models](https://mlsanity.com/paper/2609.01551/) (2026)
- [From Interpretability Methods to Interpretable Models](https://mlsanity.com/paper/2609.05399/) (2026)
- [The Polytopal Neural Network](https://mlsanity.com/paper/2610.12004/) (2026)

## Abstract (arXiv)

Vision Transformers (ViTs) are increasingly utilized in various computer vision tasks due to their powerful representation capabilities. However, it remains understudied how ViTs process information layer by layer. Numerous studies have shown that convolutional neural networks (CNNs) extract features of increasing complexity throughout their layers, which is crucial for tasks like domain adaptation and transfer learning. ViTs, lacking the same inductive biases as CNNs, can potentially learn global dependencies from the first layers due to their attention mechanisms. Given the increasing importance of ViTs in computer vision, there is a need to improve the layer-wise understanding of ViTs. In this work, we present a novel, layer-wise analysis of concepts encoded in state-of-the-art ViTs using neuron labeling. Our findings reveal that ViTs encode concepts with increasing complexity throughout the network. Early layers primarily encode basic features such as colors and textures, while later layers represent more specific classes, including objects and animals. As the complexity of encoded concepts increases, the number of concepts represented in each layer also rises, reflecting a more diverse and specific set of features. Additionally, different pretraining strategies influence the quantity and category of encoded concepts, with finetuning to specific downstream tasks generally reducing the number of encoded concepts and shifting the concepts to more relevant categories.