# Puppeteer Your Robot: Augmented Reality Leader-Follower Teleoperation

arXiv: https://arxiv.org/abs/2407.11741 · HTML: https://mlsanity.com/paper/2407.11741/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2024.

## Built on by (most cited first, 4 of 4)

- [ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback](https://mlsanity.com/paper/2410.08464/) (2024)
- [Real-Time Operator Takeover for Visuomotor Diffusion Policy Training](https://mlsanity.com/paper/2502.02308/) (2025)
- [Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation](https://mlsanity.com/paper/2605.00244/) (2026)
- [Hybrid Imitation Learning: Teleoperation Augmentation Primitives that Policies Learn to Trigger](https://mlsanity.com/paper/2512.04960/) (2025)

## Abstract (arXiv)

High-quality demonstrations are necessary when learning complex and challenging manipulation tasks. In this work, we introduce an approach to puppeteer a robot by controlling a virtual robot in an augmented reality setting. Our system allows for retaining the advantages of being intuitive from a physical leader-follower side while avoiding the unnecessary use of expensive physical setup. In addition, the user is endowed with additional information using augmented reality. We validate our system with a pilot study n=10 on a block stacking and rice scooping tasks where the majority rates the system favorably. Oculus App and corresponding ROS code are available on the project website: https://ar-puppeteer.github.io/