# Deep ViT Features as Dense Visual Descriptors

arXiv: https://arxiv.org/abs/2112.05814 · HTML: https://mlsanity.com/paper/2112.05814/

## Key facts

- Cited by 332 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 62 references that are in the map.
- First posted on arXiv 10 Dec 2021.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts](https://mlsanity.com/paper/1406.2031/) (2014)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [SPair-71k: A Large-scale Benchmark for Semantic Correspondence](https://mlsanity.com/paper/1908.10543/) (2019)
- [The Contextual Loss for Image Transformation with Non-Aligned Data](https://mlsanity.com/paper/1803.02077/) (2018)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [Interpretable and Accurate Fine-grained Recognition via Region Grouping](https://mlsanity.com/paper/2005.10411/) (2020)
- [Unsupervised Discovery of Object Landmarks as Structural Representations](https://mlsanity.com/paper/1804.04412/) (2018)
- [Unsupervised learning of object landmarks by factorized spatial embeddings](https://mlsanity.com/paper/1705.02193/) (2017)

## Built on by (most cited first, 30 of 332)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence](https://mlsanity.com/paper/2305.15347/) (2023)
- [EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision](https://mlsanity.com/paper/2311.02077/) (2023)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence](https://mlsanity.com/paper/2305.14334/) (2023)
- [StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis](https://mlsanity.com/paper/2301.09515/) (2023)
- [Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding](https://mlsanity.com/paper/2311.18482/) (2023)
- [Splicing ViT Features for Semantic Appearance Transfer](https://mlsanity.com/paper/2201.00424/) (2022)
- [Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations](https://mlsanity.com/paper/2209.03494/) (2022)
- [Probing the 3D Awareness of Visual Foundation Models](https://mlsanity.com/paper/2404.08636/) (2024)
- [SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image](https://mlsanity.com/paper/2204.00928/) (2022)
- [AnyLoc: Towards Universal Visual Place Recognition](https://mlsanity.com/paper/2308.00688/) (2023)
- [SUDS: Scalable Urban Dynamic Scenes](https://mlsanity.com/paper/2303.14536/) (2023)
- [Weakly Supervised 3D Open-vocabulary Segmentation](https://mlsanity.com/paper/2305.14093/) (2023)
- [Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation](https://mlsanity.com/paper/2308.07931/) (2023)
- [Unsupervised Semantic Correspondence Using Stable Diffusion](https://mlsanity.com/paper/2305.15581/) (2023)
- [FeatUp: A Model-Agnostic Framework for Features at Any Resolution](https://mlsanity.com/paper/2403.10516/) (2024)
- [MagicPony: Learning Articulated 3D Animals in the Wild](https://mlsanity.com/paper/2211.12497/) (2022)
- [FoundPose: Unseen Object Pose Estimation with Foundation Features](https://mlsanity.com/paper/2311.18809/) (2023)
- [textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving](https://mlsanity.com/paper/2405.20323/) (2024)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [TextDeformer: Geometry Manipulation using Text Guidance](https://mlsanity.com/paper/2304.13348/) (2023)
- [Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping](https://mlsanity.com/paper/2309.07970/) (2023)
- [FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding](https://mlsanity.com/paper/2401.01970/) (2024)

## Abstract (arXiv)

We study the use of deep features extracted from a pretrained Vision Transformer (ViT) as dense visual descriptors. We observe and empirically demonstrate that such features, when extractedfrom a self-supervised ViT model (DINO-ViT), exhibit several striking properties, including: (i) the features encode powerful, well-localized semantic information, at high spatial granularity, such as object parts; (ii) the encoded semantic information is shared across related, yet different object categories, and (iii) positional bias changes gradually throughout the layers. These properties allow us to design simple methods for a variety of applications, including co-segmentation, part co-segmentation and semantic correspondences. To distill the power of ViT features from convoluted design choices, we restrict ourselves to lightweight zero-shot methodologies (e.g., binning and clustering) applied directly to the features. Since our methods require no additional training nor data, they are readily applicable across a variety of domains. We show by extensive qualitative and quantitative evaluation that our simple methodologies achieve competitive results with recent state-of-the-art supervised methods, and outperform previous unsupervised methods by a large margin. Code is available in dino-vit-features.github.io.