# Enabling Waypoint Generation for Collaborative Robots using LLMs and Mixed Reality

arXiv: https://arxiv.org/abs/2403.09308 · HTML: https://mlsanity.com/paper/2403.09308/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2024.

## Built on by (most cited first, 3 of 3)

- [Space3D-Bench: Spatial 3D Question Answering Benchmark](https://mlsanity.com/paper/2408.16662/) (2024)
- [LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models](https://mlsanity.com/paper/2502.02441/) (2025)
- [ORCESTRA: VLM-driven Visual Robot programming in Mixed Reality](https://mlsanity.com/paper/2608.00775/) (2026)

## Abstract (arXiv)

Programming a robotic is a complex task, as it demands the user to have a good command of specific programming languages and awareness of the robot's physical constraints. We propose a framework that simplifies robot deployment by allowing direct communication using natural language. It uses large language models (LLM) for prompt processing, workspace understanding, and waypoint generation. It also employs Augmented Reality (AR) to provide visual feedback of the planned outcome. We showcase the effectiveness of our framework with a simple pick-and-place task, which we implement on a real robot. Moreover, we present an early concept of expressive robot behavior and skill generation that can be used to communicate with the user and learn new skills (e.g., object grasping).