# Robobarista: Object Part based Transfer of Manipulation Trajectories from Crowd-sourcing in 3D Pointclouds

arXiv: https://arxiv.org/abs/1504.03071 · HTML: https://mlsanity.com/paper/1504.03071/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2015.

## Built on by (most cited first, 3 of 3)

- [From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following](https://mlsanity.com/paper/1902.07742/) (2019)
- [Zero-shot Task Adaptation using Natural Language](https://mlsanity.com/paper/2106.02972/) (2021)
- [Language-guided Task Adaptation for Imitation Learning](https://mlsanity.com/paper/2301.09770/) (2023)

## Abstract (arXiv)

There is a large variety of objects and appliances in human environments, such as stoves, coffee dispensers, juice extractors, and so on. It is challenging for a roboticist to program a robot for each of these object types and for each of their instantiations. In this work, we present a novel approach to manipulation planning based on the idea that many household objects share similarly-operated object parts. We formulate the manipulation planning as a structured prediction problem and design a deep learning model that can handle large noise in the manipulation demonstrations and learns features from three different modalities: point-clouds, language and trajectory. In order to collect a large number of manipulation demonstrations for different objects, we developed a new crowd-sourcing platform called Robobarista. We test our model on our dataset consisting of 116 objects with 249 parts along with 250 language instructions, for which there are 1225 crowd-sourced manipulation demonstrations. We further show that our robot can even manipulate objects it has never seen before.