# Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations

arXiv: https://arxiv.org/abs/2306.02739 · HTML: https://mlsanity.com/paper/2306.02739/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 28 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 10 of 10)

- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [Neural Task Programming: Learning to Generalize Across Hierarchical Tasks](https://mlsanity.com/paper/1710.01813/) (2017)
- [Foundations of the Socio-physical Model of Activities (SOMA) for Autonomous Robotic Agents](https://mlsanity.com/paper/2011.11972/) (2020)
- [The Robot Household Marathon Experiment](https://mlsanity.com/paper/2011.09792/) (2020)
- [Manipulation Planning and Control for Shelf Replenishment](https://mlsanity.com/paper/1912.11018/) (2019)
- [Automated acquisition of structured, semantic models of manipulation activities from human VR demonstration](https://mlsanity.com/paper/2011.13689/) (2020)
- [Optimizing robot planning domains to reduce search time for long-horizon planning](https://mlsanity.com/paper/2111.05397/) (2021)

## Built on by (most cited first, 2 of 2)

- [RoboGrind: Intuitive and Interactive Surface Treatment with Industrial Robots](https://mlsanity.com/paper/2402.16542/) (2024)
- [Implementing Knowledge Representation and Reasoning with Object Oriented Design](https://mlsanity.com/paper/2601.14840/) (2026)

## Abstract (arXiv)

Aging societies, labor shortages and increasing wage costs call for assistance robots capable of autonomously performing a wide array of real-world tasks. Such open-ended robotic manipulation requires not only powerful knowledge representations and reasoning (KR&R) algorithms, but also methods for humans to instruct robots what tasks to perform and how to perform them. In this paper, we present a system for automatically generating executable robot control programs from human task demonstrations in virtual reality (VR). We leverage common-sense knowledge and game engine-based physics to semantically interpret human VR demonstrations, as well as an expressive and general task representation and automatic path planning and code generation, embedded into a state-of-the-art cognitive architecture. We demonstrate our approach in the context of force-sensitive fetch-and-place for a robotic shopping assistant. The source code is available at https://github.com/ease-crc/vr-program-synthesis.