# Imitation Learning-based Direct Visual Servoing using the Large Projection Formulation

arXiv: https://arxiv.org/abs/2406.09120 · HTML: https://mlsanity.com/paper/2406.09120/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 21 of 21)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [Deep Learning vs. Traditional Computer Vision](https://mlsanity.com/paper/1910.13796/) (2019)
- [Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration](https://mlsanity.com/paper/2105.06411/) (2021)
- [ImitationFlow: Learning Deep Stable Stochastic Dynamic Systems by Normalizing Flows](https://mlsanity.com/paper/2010.13129/) (2020)
- [One-Shot Imitation Learning: A Pose Estimation Perspective](https://mlsanity.com/paper/2310.12077/) (2023)
- [KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation](https://mlsanity.com/paper/2007.13960/) (2020)
- [Siamese Convolutional Neural Network for Sub-millimeter-accurate Camera Pose Estimation and Visual Servoing](https://mlsanity.com/paper/1903.04713/) (2019)
- [Towards Orientation Learning and Adaptation in Cartesian Space](https://mlsanity.com/paper/1907.03918/) (2019)
- [Merging Position and Orientation Motion Primitives](https://mlsanity.com/paper/2003.11507/) (2020)
- [Continual Learning from Demonstration of Robotics Skills](https://mlsanity.com/paper/2202.06843/) (2022)
- [Exploring Convolutional Networks for End-to-End Visual Servoing](https://mlsanity.com/paper/1706.03220/) (2017)
- [Uncertainty-Aware Self-Supervised Learning of Spatial Perception Tasks](https://mlsanity.com/paper/2103.12007/) (2021)
- [Scalable and Efficient Continual Learning from Demonstration via a Hypernetwork-generated Stable Dynamics Model](https://mlsanity.com/paper/2311.03600/) (2023)
- [Learning Deep Robotic Skills on Riemannian manifolds](https://mlsanity.com/paper/2210.15244/) (2022)
- [Learning Stable Dynamical Systems for Visual Servoing](https://mlsanity.com/paper/2204.05681/) (2022)
- [Visual Servoing with Geometrically Interpretable Neural Perception](https://mlsanity.com/paper/2210.10549/) (2022)
- [Imitation Learning-based Visual Servoing for Tracking Moving Objects](https://mlsanity.com/paper/2309.07729/) (2023)

## Abstract (arXiv)

Today robots must be safe, versatile, and user-friendly to operate in unstructured and human-populated environments. Dynamical system-based imitation learning enables robots to perform complex tasks stably and without explicit programming, greatly simplifying their real-world deployment. To exploit the full potential of these systems it is crucial to implement closed loops that use visual feedback. Vision permits to cope with environmental changes, but is complex to handle due to the high dimension of the image space. This study introduces a dynamical system-based imitation learning for direct visual servoing. It leverages off-the-shelf deep learning-based perception modules to extract robust features from the raw input image, and an imitation learning strategy to execute sophisticated robot motions. The learning blocks are integrated using the large projection task priority formulation. As demonstrated through extensive experimental analysis, the proposed method realizes complex tasks with a robotic manipulator.