mlsanityData to 7 Oct 2026

Imitation Learning-based Direct Visual Servoing using the Large Projection Formulation

arXiv 2406.09120 ↗First posted 13 Jun 2024cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Today robots must be safe, versatile, and user-friendly to operate in unstructured and human-populated environments. Dynamical system-based imitation learning enables robots to perform complex tasks stably and without explicit programming, greatly simplifying their real-world deployment. To exploit the full potential of these systems it is crucial to implement closed loops that use visual feedback. Vision permits to cope with environmental changes, but is complex to handle due to the high dimension of the image space. This study introduces a dynamical system-based imitation learning for direct visual servoing. It leverages off-the-shelf deep learning-based perception modules to extract robust features from the raw input image, and an imitation learning strategy to execute sophisticated robot motions. The learning blocks are integrated using the large projection task priority formulation. As demonstrated through extensive experimental analysis, the proposed method realizes complex tasks with a robotic manipulator.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 50 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

21 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023One-Shot Imitation Learning: A Pose Estimation Perspective18 Oct
  2. 2023Imitation Learning-based Visual Servoing for Tracking Moving Objects14 Sep
  3. 2022Learning Stable Dynamical Systems for Visual Servoing12 Apr
  4. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration13 May

Other references · 17

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  4. 2018DeepIM: Deep Iterative Matching for 6D Pose Estimationcited by 283
  5. 2019Self-supervised 6D Object Pose Estimation for Robot Manipulationcited by 100
  6. 2019Deep Learning vs. Traditional Computer Visioncited by 94
  7. 2020ImitationFlow: Learning Deep Stable Stochastic Dynamic Systems by Normalizing Flowscited by 33
  8. 2020KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulationcited by 16
  9. 2019Siamese Convolutional Neural Network for Sub-millimeter-accurate Camera Pose Estimation and Visual Servoingcited by 12
  10. 2019Towards Orientation Learning and Adaptation in Cartesian Spacecited by 12
  11. 2020Merging Position and Orientation Motion Primitivescited by 12
  12. 2022Continual Learning from Demonstration of Robotics Skillscited by 11
5 more
  1. 2017Exploring Convolutional Networks for End-to-End Visual Servoingcited by 10
  2. 2021Uncertainty-Aware Self-Supervised Learning of Spatial Perception Taskscited by 8
  3. 2023Scalable and Efficient Continual Learning from Demonstration via a Hypernetwork-generated Stable Dynamics Modelcited by 6
  4. 2022Learning Deep Robotic Skills on Riemannian manifoldscited by 5
  5. 2022Visual Servoing with Geometrically Interpretable Neural Perceptioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Visual Imitation Learning of Non-Prehensile Manipulation Tasks with Dynamics-Supervised Models
  2. 2025VGGT-DP: Generalizable Robot Control via Vision Foundation Models
  3. 2026Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning
  4. 2024Fusion Dynamical Systems with Machine Learning in Imitation Learning: A Comprehensive Overview
  5. 2024Imagine2Servo: Intelligent Visual Servoing with Diffusion-Driven Goal Generation for Robotic Tasks
  6. 2023Robust Visual Imitation Learning with Inverse Dynamics Representations
  7. 2024VIEW: Visual Imitation Learning with Waypoints
  8. 2026DiffusionVS: A Generative Framework for Robust Visual Servoing Based on Diffusion Policy