# Interpretable Vision Transformers in Image Classification via SVDA

arXiv: https://arxiv.org/abs/2602.10994 · HTML: https://mlsanity.com/paper/2602.10994/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [cosFormer: Rethinking Softmax in Attention](https://mlsanity.com/paper/2202.08791/) (2022)
- [Specformer: Spectral Graph Neural Networks Meet Transformers](https://mlsanity.com/paper/2303.01028/) (2023)
- [Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation](https://mlsanity.com/paper/2305.19798/) (2023)
- [Transformer-VQ: Linear-Time Transformers via Vector Quantization](https://mlsanity.com/paper/2309.16354/) (2023)
- [Interpretable Image Classification with Adaptive Prototype-based Vision Transformers](https://mlsanity.com/paper/2410.20722/) (2024)
- [On the Faithfulness of Vision Transformer Explanations](https://mlsanity.com/paper/2404.01415/) (2024)
- [Interpretability-Aware Vision Transformer](https://mlsanity.com/paper/2309.08035/) (2023)

## Built on by (most cited first, 1 of 1)

- [Prescriptive SVD-Inspired Attention via Spectral Energy Retention](https://mlsanity.com/paper/2609.24370/) (2026)

## Abstract (arXiv)

Vision Transformers (ViTs) have achieved state-of-the-art performance in image classification, yet their attention mechanisms often remain opaque and exhibit dense, non-structured behaviors. In this work, we adapt our previously proposed SVD-Inspired Attention (SVDA) mechanism to the ViT architecture, introducing a geometrically grounded formulation that enhances interpretability, sparsity, and spectral structure. We apply the use of interpretability indicators -- originally proposed with SVDA -- to monitor attention dynamics during training and assess structural properties of the learned representations. Experimental evaluations on four widely used benchmarks -- CIFAR-10, FashionMNIST, CIFAR-100, and ImageNet-100 -- demonstrate that SVDA consistently yields more interpretable attention patterns without sacrificing classification accuracy. While the current framework offers descriptive insights rather than prescriptive guidance, our results establish SVDA as a comprehensive and informative tool for analyzing and developing structured attention models in computer vision. This work lays the foundation for future advances in explainable AI, spectral diagnostics, and attention-based model compression.