# WM-VS: Progress-Aligned World Models for Closed-Loop Visual Servoing

arXiv: https://arxiv.org/abs/2609.20892 · HTML: https://mlsanity.com/paper/2609.20892/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Deep Visual MPC-Policy Learning for Navigation](https://mlsanity.com/paper/1903.02749/) (2019)
- [KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation](https://mlsanity.com/paper/2007.13960/) (2020)
- [Siamese Convolutional Neural Network for Sub-millimeter-accurate Camera Pose Estimation and Visual Servoing](https://mlsanity.com/paper/1903.04713/) (2019)
- [Exploring Convolutional Networks for End-to-End Visual Servoing](https://mlsanity.com/paper/1706.03220/) (2017)
- [Learning Visual Servoing with Deep Features and Fitted Q-Iteration](https://mlsanity.com/paper/1703.11000/) (2017)
- [Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views](https://mlsanity.com/paper/2511.12878/) (2025)
- [DFBVS: Deep Feature-Based Visual Servo](https://mlsanity.com/paper/2201.08046/) (2022)
- [DeepMPCVS: Deep Model Predictive Control for Visual Servoing](https://mlsanity.com/paper/2105.00788/) (2021)
- [DFVS: Deep Flow Guided Scene Agnostic Image Based Visual Servoing](https://mlsanity.com/paper/2003.03766/) (2020)
- [ViT-VS: On the Applicability of Pretrained Vision Transformer Features for Generalizable Visual Servoing](https://mlsanity.com/paper/2503.04545/) (2025)
- [Self-supervised deep visual servoing for high precision peg-in-hole insertion](https://mlsanity.com/paper/2206.08800/) (2022)

## Abstract (arXiv)

Closed-loop visual servoing requires predictions that indicate whether an action reduces task error, not only whether the action is plausible. We call this gap the prediction-control mismatch and introduce WM-VS, a target-centric progress-aligned world-model framework for closed-loop visual servoing. Offline target-region DINOv2 correspondences define a signed four-dimensional servo coordinate for translation, scale, and in-plane rotation. Stage 1 aligns action-conditioned latent transitions with this coordinate; Stage 2 freezes the world model and trains a reactive joint-velocity policy with action imitation, consequence supervision, and short imagined rollouts that favor error contraction. Deployment is RGB-only and reactive, without online trajectory optimization. On a real 7-DoF eye-to-hand system, WM-VS reaches a corner RMSE no larger than 10 percent of its initial value in 30/30 trials and retains this criterion at the final valid frame in 25/30 (83.33 percent). Removing future-error alignment reduces retention to 26.67 percent. The learned progress signal agrees with an external AprilTag corner error not used for training or control (mean Spearman rho = 0.8778). Without retraining, two unseen 3D targets achieve translation-error reductions of 86.48 percent and 90.27 percent and rotation-error reductions of 70.01 percent and 65.70 percent. These results link progress-aligned action consequences to repeated closed-loop correction and transfer. Code and data will be released as open source.