# RAMPA: Robotic Augmented Reality for Machine Programming by DemonstrAtion

arXiv: https://arxiv.org/abs/2410.13412 · HTML: https://mlsanity.com/paper/2410.13412/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 11 of 11)

- [Real-World Robot Applications of Foundation Models: A Review](https://mlsanity.com/paper/2402.05741/) (2024)
- [Holo-Dex: Teaching Dexterity with Immersive Mixed Reality](https://mlsanity.com/paper/2210.06463/) (2022)
- [Dynamic Movement Primitives in Robotics: A Tutorial Survey](https://mlsanity.com/paper/2102.03861/) (2021)
- [ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback](https://mlsanity.com/paper/2410.08464/) (2024)
- [CoSTAR: Instructing Collaborative Robots with Behavior Trees and Vision](https://mlsanity.com/paper/1611.06145/) (2016)
- [Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces](https://mlsanity.com/paper/2203.03254/) (2022)
- [Augmented Object Intelligence with XR-Objects](https://mlsanity.com/paper/2404.13274/) (2024)
- [InfraredTags: Embedding Invisible AR Markers and Barcodes Using Low-Cost, Infrared-Based 3D Printing and Imaging Tools](https://mlsanity.com/paper/2202.06165/) (2022)
- [Communicating Robot Arm Motion Intent Through Mixed Reality Head-mounted Displays](https://mlsanity.com/paper/1708.03655/) (2017)
- [Virtual, Augmented, and Mixed Reality for Human-Robot Interaction: A Survey and Virtual Design Element Taxonomy](https://mlsanity.com/paper/2202.11249/) (2022)
- [Conditional Neural Expert Processes for Learning Movement Primitives from Demonstration](https://mlsanity.com/paper/2402.08424/) (2024)

## Abstract (arXiv)

This paper introduces Robotic Augmented Reality for Machine Programming by Demonstration (RAMPA), the first ML-integrated, XR-driven end-to-end robotic system, allowing training and deployment of ML models such as ProMPs on the fly, and utilizing the capabilities of state-of-the-art and commercially available AR headsets, e.g., Meta Quest 3, to facilitate the application of Programming by Demonstration (PbD) approaches on industrial robotic arms, e.g., Universal Robots UR10. Our approach enables in-situ data recording, visualization, and fine-tuning of skill demonstrations directly within the user's physical environment. RAMPA addresses critical challenges of PbD, such as safety concerns, programming barriers, and the inefficiency of collecting demonstrations on the actual hardware. The performance of our system is evaluated against the traditional method of kinesthetic control in teaching three different robotic manipulation tasks and analyzed with quantitative metrics, measuring task performance and completion time, trajectory smoothness, system usability, user experience, and task load using standardized surveys. Our findings indicate a substantial advancement in how robotic tasks are taught and refined, promising improvements in operational safety, efficiency, and user engagement in robotic programming.