# Situated Multimodal Control of a Mobile Robot: Navigation through a Virtual Environment

arXiv: https://arxiv.org/abs/2007.09053 · HTML: https://mlsanity.com/paper/2007.09053/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 1 of 1)

- [An Abstract Specification of VoxML as an Annotation Language](https://mlsanity.com/paper/2305.13076/) (2023)

## Abstract (arXiv)

We present a new interface for controlling a navigation robot in novel environments using coordinated gesture and language. We use a TurtleBot3 robot with a LIDAR and a camera, an embodied simulation of what the robot has encountered while exploring, and a cross-platform bridge facilitating generic communication. A human partner can deliver instructions to the robot using spoken English and gestures relative to the simulated environment, to guide the robot through navigation tasks.