# Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives

arXiv: https://arxiv.org/abs/2003.08854 · HTML: https://mlsanity.com/paper/2003.08854/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2020.

## Built on by (most cited first, 7 of 7)

- [GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation](https://mlsanity.com/paper/2410.06158/) (2024)
- [Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers](https://mlsanity.com/paper/2403.12943/) (2024)
- [Semantically Grounded Object Matching for Robust Robotic Scene Rearrangement](https://mlsanity.com/paper/2111.07975/) (2021)
- [Building Open-Ended Embodied Agent via Language-Policy Bidirectional Adaptation](https://mlsanity.com/paper/2401.00006/) (2023)
- [Efficient Skill Acquisition for Complex Manipulation Tasks in Obstructed Environments](https://mlsanity.com/paper/2303.03365/) (2023)
- [Less is more -- the Dispatcher/ Executor principle for multi-task Reinforcement Learning](https://mlsanity.com/paper/2312.09120/) (2023)
- [Introspective Visuomotor Control: Exploiting Uncertainty in Deep Visuomotor Control for Failure Recovery](https://mlsanity.com/paper/2103.11881/) (2021)

## Abstract (arXiv)

Visuomotor control (VMC) is an effective means of achieving basic manipulation tasks such as pushing or pick-and-place from raw images. Conditioning VMC on desired goal states is a promising way of achieving versatile skill primitives. However, common conditioning schemes either rely on task-specific fine tuning - e.g. using one-shot imitation learning (IL) - or on sampling approaches using a forward model of scene dynamics i.e. model-predictive control (MPC), leaving deployability and planning horizon severely limited. In this paper we propose a conditioning scheme which avoids these pitfalls by learning the controller and its conditioning in an end-to-end manner. Our model predicts complex action sequences based directly on a dynamic image representation of the robot motion and the distance to a given target observation. In contrast to related works, this enables our approach to efficiently perform complex manipulation tasks from raw image observations without predefined control primitives or test time demonstrations. We report significant improvements in task success over representative MPC and IL baselines. We also demonstrate our model's generalisation capabilities in challenging, unseen tasks featuring visual noise, cluttered scenes and unseen object geometries.