# KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation

arXiv: https://arxiv.org/abs/2007.13960 · HTML: https://mlsanity.com/paper/2007.13960/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2020.

## Built on by (most cited first, 16 of 16)

- [Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration](https://mlsanity.com/paper/2105.06411/) (2021)
- [Vision-Based Manipulators Need to Also See from Their Hands](https://mlsanity.com/paper/2203.12677/) (2022)
- [Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces](https://mlsanity.com/paper/2105.11283/) (2021)
- [Learning Eye-in-Hand Camera Calibration from a Single Image](https://mlsanity.com/paper/2111.01245/) (2021)
- [One-Shot Dual-Arm Imitation Learning](https://mlsanity.com/paper/2503.06831/) (2025)
- [End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation](https://mlsanity.com/paper/2202.06027/) (2022)
- [CNSv2: Probabilistic Correspondence Encoded Neural Image Servo](https://mlsanity.com/paper/2503.00132/) (2025)
- [ViT-VS: On the Applicability of Pretrained Vision Transformer Features for Generalizable Visual Servoing](https://mlsanity.com/paper/2503.04545/) (2025)
- [Self-supervised Wide Baseline Visual Servoing via 3D Equivariance](https://mlsanity.com/paper/2209.05432/) (2022)
- [Improving Object Permanence using Agent Actions and Reasoning](https://mlsanity.com/paper/2110.00238/) (2021)
- [Learning A Simulation-based Visual Policy for Real-world Peg In Unseen Holes](https://mlsanity.com/paper/2205.04297/) (2022)
- [Tracking Object Positions in Reinforcement Learning: A Metric for Keypoint Detection (extended version)](https://mlsanity.com/paper/2312.00592/) (2023)
- [A Closed-Loop Multi-perspective Visual Servoing Approach with Reinforcement Learning](https://mlsanity.com/paper/2312.15809/) (2023)
- [Imitation Learning-based Direct Visual Servoing using the Large Projection Formulation](https://mlsanity.com/paper/2406.09120/) (2024)
- [WM-VS: Progress-Aligned World Models for Closed-Loop Visual Servoing](https://mlsanity.com/paper/2609.20892/) (2026)
- [VGM-VS: Rethinking Visual Geometry Model for High-Precision Visual Servoing](https://mlsanity.com/paper/2609.28312/) (2026)

## Abstract (arXiv)

We present KOVIS, a novel learning-based, calibration-free visual servoing method for fine robotic manipulation tasks with eye-in-hand stereo camera system. We train the deep neural network only in the simulated environment; and the trained model could be directly used for real-world visual servoing tasks. KOVIS consists of two networks. The first keypoint network learns the keypoint representation from the image using with an autoencoder. Then the visual servoing network learns the motion based on keypoints extracted from the camera image. The two networks are trained end-to-end in the simulated environment by self-supervised learning without manual data labeling. After training with data augmentation, domain randomization, and adversarial examples, we are able to achieve zero-shot sim-to-real transfer to real-world robotic manipulation tasks. We demonstrate the effectiveness of the proposed method in both simulated environment and real-world experiment with different robotic manipulation tasks, including grasping, peg-in-hole insertion with 4mm clearance, and M13 screw insertion. The demo video is available at http://youtu.be/gfBJBR2tDzA