# LLM-Driven Augmented Reality Puppeteer: Controller-Free Voice-Commanded Robot Teleoperation

arXiv: https://arxiv.org/abs/2502.09142 · HTML: https://mlsanity.com/paper/2502.09142/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2025.

## Built on by (most cited first, 4 of 4)

- [A Study of the Framework and Real-World Applications of Language Embedding for 3D Scene Understanding](https://mlsanity.com/paper/2508.05064/) (2025)
- [Voice-Interactive Surgical Agent for Multimodal Patient Data Control](https://mlsanity.com/paper/2511.07392/) (2025)
- [Hybrid Imitation Learning: Teleoperation Augmentation Primitives that Policies Learn to Trigger](https://mlsanity.com/paper/2512.04960/) (2025)
- [Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining](https://mlsanity.com/paper/2602.20500/) (2026)

## Abstract (arXiv)

The integration of robotics and augmented reality (AR) presents transformative opportunities for advancing human-robot interaction (HRI) by improving usability, intuitiveness, and accessibility. This work introduces a controller-free, LLM-driven voice-commanded AR puppeteering system, enabling users to teleoperate a robot by manipulating its virtual counterpart in real time. By leveraging natural language processing (NLP) and AR technologies, our system -- prototyped using Meta Quest 3 -- eliminates the need for physical controllers, enhancing ease of use while minimizing potential safety risks associated with direct robot operation. A preliminary user demonstration successfully validated the system's functionality, demonstrating its potential for safer, more intuitive, and immersive robotic control.