# VGM-VS: Rethinking Visual Geometry Model for High-Precision Visual Servoing

arXiv: https://arxiv.org/abs/2609.28312 · HTML: https://mlsanity.com/paper/2609.28312/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [π3: Permutation-Equivariant Visual Geometry Learning](https://mlsanity.com/paper/2507.13347/) (2025)
- [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025)
- [3D Reconstruction with Spatial Memory](https://mlsanity.com/paper/2408.16061/) (2024)
- [Geometric Loss Functions for Camera Pose Regression with Deep Learning](https://mlsanity.com/paper/1704.00390/) (2017)
- [MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors](https://mlsanity.com/paper/2412.12392/) (2024)
- [OmniVGGT: Omni-Modality Driven Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2511.10560/) (2025)
- [KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation](https://mlsanity.com/paper/2007.13960/) (2020)
- [Siamese Convolutional Neural Network for Sub-millimeter-accurate Camera Pose Estimation and Visual Servoing](https://mlsanity.com/paper/1903.04713/) (2019)
- [GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulation](https://mlsanity.com/paper/2509.15733/) (2025)
- [VGGT-DP: Generalizable Robot Control via Vision Foundation Models](https://mlsanity.com/paper/2509.18778/) (2025)
- [Exploring Convolutional Networks for End-to-End Visual Servoing](https://mlsanity.com/paper/1706.03220/) (2017)
- [Reloc-VGGT: Visual Re-localization with Geometry Grounded Transformer](https://mlsanity.com/paper/2512.21883/) (2025)
- [DFBVS: Deep Feature-Based Visual Servo](https://mlsanity.com/paper/2201.08046/) (2022)
- [DeepMPCVS: Deep Model Predictive Control for Visual Servoing](https://mlsanity.com/paper/2105.00788/) (2021)
- [CNS: Correspondence Encoded Neural Image Servo Policy](https://mlsanity.com/paper/2309.09047/) (2023)
- [DFVS: Deep Flow Guided Scene Agnostic Image Based Visual Servoing](https://mlsanity.com/paper/2003.03766/) (2020)
- [CNSv2: Probabilistic Correspondence Encoded Neural Image Servo](https://mlsanity.com/paper/2503.00132/) (2025)
- [ViT-VS: On the Applicability of Pretrained Vision Transformer Features for Generalizable Visual Servoing](https://mlsanity.com/paper/2503.04545/) (2025)
- [ART-VS: Adaptive Resolution Tiling for Vision Transformer Visual Servoing](https://mlsanity.com/paper/2606.19089/) (2026)

## Abstract (arXiv)

We present VGM-VS, a visual servoing method built on a pretrained feed-forward visual geometry model. Given the current view and a reference image captured at the target configuration, we estimate the relative camera pose with a visual geometry model and apply it iteratively as the pose increment of a closed-loop pose-based visual servoing (PBVS) scheme. The geometry-aware representation acquired from large-scale pretraining keeps this estimate reliable when the target is occluded, weakly textured, or covers only a small part of the image. However, the scale ambiguity inherent to these models leaves the predicted translation defined up to an unknown scale, while the pose increment must be metric for robot control. We close this gap with a scene-specific metric adaptation: the robot autonomously records image--pose pairs along a predefined motion starting from the target pose, and we fine-tune the camera head on these data, jointly learning the hand--eye transform and thus removing the need for a dedicated calibration process. We evaluate our method on three real-world assembly tasks with demanding tolerances: USB-C cable picking, cable insertion, and RAM insertion. Running in real time at 30Hz, VGM-VS converges to submillimeter terminal accuracy on the cable tasks, and reaches success rates of 90--100% when the target is moved during servoing. It converges in all trials under initial displacements of up to 30cm from the reference pose and with 50% of the target object occluded, outperforming the compared visual servoing baselines.