mlsanityData to 14 Oct 2026

Vision Transformers Need Registers

arXiv 2309.16588 ↗First posted 28 Sep 2023cs.CVcited by 871

Not on the map. Why?

Cited by 107 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers have recently emerged as a powerful tool for learning visual representations. In this paper, we identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative background areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input sequence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervised models, sets a new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads to smoother feature maps and attention maps for downstream visual processing.

Key facts. Cited by 871 papers in the map (data through 14 Oct 2026). Builds on 29 of its 34 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

29 of 34 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023Segment Anythingcited by 8,234
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
17 more
  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  2. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  3. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  4. 2020Object-Centric Learning with Slot Attentioncited by 876
  5. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  6. 2022DeiT III: Revenge of the ViTcited by 419
  7. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  8. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 230
  9. 2022Recurrent Memory Transformercited by 203
  10. 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 187
  11. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  12. 2020Memory Transformercited by 76
  13. 2021ViDT: An Efficient and Effective Fully Transformer-based Object Detectorcited by 46
  14. 2022Fine-tuning Image Transformers using Learnable Memorycited by 34
  15. 2023Emergence of Segmentation with Minimalistic White-Box Transformerscited by 24
  16. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  17. 2023Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?cited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 193; 2025: 323; 2026: 342.

871 papers.

Sort:

By area:

  1. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  2. 2024Depth Anything V2cited by 1,422
  3. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  4. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 395
  5. 2024What matters when building vision-language models?cited by 326
  6. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  7. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  8. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  9. 2024Massive Activations in Large Language Modelscited by 294
  10. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  11. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  12. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
38 more
  1. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  2. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
  3. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  4. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  5. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
  6. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
  7. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  8. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  9. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  10. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  11. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 161
  12. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 157
  13. 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 143
  14. 2025Training Agents Inside of Scalable World Modelscited by 136
  15. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  16. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 132
  17. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
  18. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 120
  19. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  20. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  21. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
  22. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 111
  23. 2025Sonata: Self-Supervised Learning of Reliable Point Representationscited by 110
  24. 2023FoundPose: Unseen Object Pose Estimation with Foundation Featurescited by 109
  25. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  26. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  27. 2024Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detectioncited by 93
  28. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 91
  29. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 89
  30. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 86
  31. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  32. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  33. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 73
  34. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 71
  35. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 71
  36. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  37. 2024SINGAPO: Single Image Controlled Generation of Articulated Parts in Objectscited by 69
  38. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 67

50 most cited of 871 All 871 (JSON)