# SVD-ViT: Does SVD Make Vision Transformers Attend More to the Foreground?

arXiv: https://arxiv.org/abs/2602.02765 · HTML: https://mlsanity.com/paper/2602.02765/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Builds on (most cited first, 9 of 9)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [Mamba-R: Vision Mamba ALSO Needs Registers](https://mlsanity.com/paper/2405.14858/) (2024)
- [Vision Transformers Don't Need Trained Registers](https://mlsanity.com/paper/2506.08010/) (2025)
- [SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learning](https://mlsanity.com/paper/2412.04077/) (2024)
- [SALT: Parameter-Efficient Fine-Tuning via Singular Value Adaptation with Low-Rank Transformation](https://mlsanity.com/paper/2503.16055/) (2025)

## Built on by (most cited first, 1 of 1)

- [Muon in Vision Transformers: Optimizer-Recipe Interactions and Gradient Spectra](https://mlsanity.com/paper/2605.24770/) (2026)

## Abstract (arXiv)

Vision Transformers (ViT) have been established as large-scale foundation models. However, because self-attention operates globally, they lack an explicit mechanism to distinguish foreground from background. As a result, ViT may learn unnecessary background features and artifacts, leading to degraded classification performance. To address this issue, we propose SVD-ViT, which leverages singular value decomposition (SVD) to prioritize the learning of foreground features. SVD-ViT consists of three components-SPC module, SSVA, and ID-RSVD-and suppresses task-irrelevant factors such as background noise and artifacts by extracting and aggregating singular vectors that capture object foreground information. Experimental results demonstrate that our method improves classification accuracy and effectively learns informative foreground representations while reducing the impact of background noise.