# Sim2Real View Invariant Visual Servoing by Recurrent Control

arXiv: https://arxiv.org/abs/1712.07642 · HTML: https://mlsanity.com/paper/1712.07642/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2017.

## Built on by (most cited first, 19 of 19)

- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Scaling Robot Learning with Semantically Imagined Experience](https://mlsanity.com/paper/2302.11550/) (2023)
- [From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence](https://mlsanity.com/paper/2110.15245/) (2021)
- [Single-view robot pose and joint angle estimation via render & compare](https://mlsanity.com/paper/2104.09359/) (2021)
- [Polybot: Training One Policy Across Robots While Embracing Variability](https://mlsanity.com/paper/2307.03719/) (2023)
- [DROPO: Sim-to-Real Transfer with Offline Domain Randomization](https://mlsanity.com/paper/2201.08434/) (2022)
- [Robust Hybrid Learning With Expert Augmentation](https://mlsanity.com/paper/2202.03881/) (2022)
- [Model-based Behavioral Cloning with Future Image Similarity Learning](https://mlsanity.com/paper/1910.03157/) (2019)
- [Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Tasks](https://mlsanity.com/paper/2303.07026/) (2023)
- [RoboTransfer: Controllable Geometry-Consistent Video Diffusion for Manipulation Policy Transfer](https://mlsanity.com/paper/2505.23171/) (2025)
- [End-to-End Egospheric Spatial Memory](https://mlsanity.com/paper/2102.07764/) (2021)
- [Validate on Sim, Detect on Real -- Model Selection for Domain Randomization](https://mlsanity.com/paper/2111.00765/) (2021)
- [Visual Servoing for Pose Control of Soft Continuum Arm in a Structured Environment](https://mlsanity.com/paper/2202.05200/) (2022)
- [Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)](https://mlsanity.com/paper/2509.06191/) (2025)
- [Hierarchical Policy for Non-prehensile Multi-object Rearrangement with Deep Reinforcement Learning and Monte Carlo Tree Search](https://mlsanity.com/paper/2109.08973/) (2021)
- [FOD-S2R: A FOD Dataset for Sim2Real Transfer Learning based Object Detection](https://mlsanity.com/paper/2512.01315/) (2025)
- [Nonprehensile Riemannian Motion Predictive Control](https://mlsanity.com/paper/2111.07986/) (2021)
- [AutoDIME: Automatic Design of Interesting Multi-Agent Environments](https://mlsanity.com/paper/2203.02481/) (2022)
- [Sim-Anchored Learning for On-the-Fly Adaptation](https://mlsanity.com/paper/2301.06987/) (2023)

## Abstract (arXiv)

Humans are remarkably proficient at controlling their limbs and tools from a wide range of viewpoints and angles, even in the presence of optical distortions. In robotics, this ability is referred to as visual servoing: moving a tool or end-point to a desired location using primarily visual feedback. In this paper, we study how viewpoint-invariant visual servoing skills can be learned automatically in a robotic manipulation scenario. To this end, we train a deep recurrent controller that can automatically determine which actions move the end-point of a robotic arm to a desired object. The problem that must be solved by this controller is fundamentally ambiguous: under severe variation in viewpoint, it may be impossible to determine the actions in a single feedforward operation. Instead, our visual servoing system must use its memory of past movements to understand how the actions affect the robot motion from the current viewpoint, correcting mistakes and gradually moving closer to the target. This ability is in stark contrast to most visual servoing methods, which either assume known dynamics or require a calibration phase. We show how we can learn this recurrent controller using simulated data and a reinforcement learning objective. We then describe how the resulting model can be transferred to a real-world robot by disentangling perception from control and only adapting the visual layers. The adapted model can servo to previously unseen objects from novel viewpoints on a real-world Kuka IIWA robotic arm. For supplementary videos, see: https://fsadeghi.github.io/Sim2RealViewInvariantServo