# Vision Transformers Need More Than Registers

arXiv: https://arxiv.org/abs/2602.22394 · HTML: https://mlsanity.com/paper/2602.22394/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 60 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Open-Vocabulary DETR with Conditional Matching](https://mlsanity.com/paper/2203.11876/) (2022)
- [SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference](https://mlsanity.com/paper/2312.01597/) (2023)
- [F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models](https://mlsanity.com/paper/2209.15639/) (2022)
- [Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection](https://mlsanity.com/paper/2207.03482/) (2022)
- [CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching](https://mlsanity.com/paper/2303.13076/) (2023)

## Built on by (most cited first, 15 of 15)

- [VGGT-Omega](https://mlsanity.com/paper/2605.15195/) (2026)
- [Registers Matter for Pixel-Space Diffusion Transformers](https://mlsanity.com/paper/2605.16147/) (2026)
- [NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving](https://mlsanity.com/paper/2605.31116/) (2026)
- [LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute](https://mlsanity.com/paper/2605.06809/) (2026)
- [Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models](https://mlsanity.com/paper/2605.07148/) (2026)
- [Zero-Ablation Overstates Register Content Dependence in DINO Vision Transformers](https://mlsanity.com/paper/2604.14433/) (2026)
- [SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models](https://mlsanity.com/paper/2605.07800/) (2026)
- [What-Where Transformer: A Slot-Centric Visual Backbone for Concurrent Representation and Localization](https://mlsanity.com/paper/2605.12021/) (2026)
- [When Dynamics Models Read the Wrong Time Steps: Label-Free Event Credit Re-Anchoring for Robust Global Readouts](https://mlsanity.com/paper/2606.17572/) (2026)
- [More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning](https://mlsanity.com/paper/2607.09825/) (2026)
- [Feed-Forward Hierarchical Gaussian Diffusion for Extreme CT Reconstruction](https://mlsanity.com/paper/2608.15343/) (2026)
- [Beyond Classification: Structured Supervision Aligns Visual Evidence with Medical Semantics](https://mlsanity.com/paper/2609.05937/) (2026)
- [CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework](https://mlsanity.com/paper/2609.17026/) (2026)
- [OFBD: Object-Focused Background Debiasing for Long-Tailed Learning](https://mlsanity.com/paper/2609.37331/) (2026)
- [Learning Representations to Guide Pixel Diffusion Transformers](https://mlsanity.com/paper/2610.12978/) (2026)

## Abstract (arXiv)

Vision Transformers (ViTs), when pre-trained on large-scale data, provide general-purpose representations for diverse downstream tasks. However, artifacts in ViTs are widely observed across different supervision paradigms and downstream tasks. Through systematic analysis of artifacts in ViTs, we find that their fundamental mechanisms have yet to be sufficiently elucidated. In this paper, through systematic analysis, we conclude that these artifacts originate from a lazy aggregation behavior: ViT uses semantically irrelevant background patches as shortcuts to represent global semantics, driven by global attention and Coarse-grained semantic supervision. Our solution selectively integrates patch features into the CLS token, reducing the influence of background-dominated shortcuts and consistently improving performance across 12 benchmarks under label-, text-, and self-supervision. We hope this work offers a new perspective on ViT behavior.