# ViT-VS: On the Applicability of Pretrained Vision Transformer Features for Generalizable Visual Servoing

arXiv: https://arxiv.org/abs/2503.04545 · HTML: https://mlsanity.com/paper/2503.04545/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Builds on (most cited first, 12 of 12)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence](https://mlsanity.com/paper/2311.17034/) (2023)
- [Zero-Shot Category-Level Object Pose Estimation](https://mlsanity.com/paper/2204.03635/) (2022)
- [Image Matching Using SIFT, SURF, BRIEF and ORB: Performance Comparison for Distorted Images](https://mlsanity.com/paper/1710.02726/) (2017)
- [KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation](https://mlsanity.com/paper/2007.13960/) (2020)
- [DFBVS: Deep Feature-Based Visual Servo](https://mlsanity.com/paper/2201.08046/) (2022)
- [Dual Quaternion-Based Visual Servoing for Grasping Moving Objects](https://mlsanity.com/paper/2107.08149/) (2021)
- [Self-supervised Wide Baseline Visual Servoing via 3D Equivariance](https://mlsanity.com/paper/2209.05432/) (2022)

## Built on by (most cited first, 2 of 2)

- [WM-VS: Progress-Aligned World Models for Closed-Loop Visual Servoing](https://mlsanity.com/paper/2609.20892/) (2026)
- [VGM-VS: Rethinking Visual Geometry Model for High-Precision Visual Servoing](https://mlsanity.com/paper/2609.28312/) (2026)

## Abstract (arXiv)

Visual servoing enables robots to precisely position their end-effector relative to a target object. While classical methods rely on hand-crafted features and thus are universally applicable without task-specific training, they often struggle with occlusions and environmental variations, whereas learning-based approaches improve robustness but typically require extensive training. We present a visual servoing approach that leverages pretrained vision transformers for semantic feature extraction, combining the advantages of both paradigms while also being able to generalize beyond the provided sample. Our approach achieves full convergence in unperturbed scenarios and surpasses classical image-based visual servoing by up to 31.2% relative improvement in perturbed scenarios. Even the convergence rates of learning-based methods are matched despite requiring no task- or object-specific training. Real-world evaluations confirm robust performance in end-effector positioning, industrial box manipulation, and grasping of unseen objects using only a reference from the same category. Our code and simulation environment are available at: https://alessandroscherl.github.io/ViT-VS/