# Augmented Reality for RObots (ARRO): Pointing Visuomotor Policies Towards Visual Robustness

arXiv: https://arxiv.org/abs/2505.08627 · HTML: https://mlsanity.com/paper/2505.08627/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2025.

## Built on by (most cited first, 8 of 8)

- [PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies](https://mlsanity.com/paper/2509.18282/) (2025)
- [Cloak: Zero-Shot Cross-Embodiment Manipulation by Masking the End-Effector from the VLA](https://mlsanity.com/paper/2606.22836/) (2026)
- [VLAgents: A Policy Server for Efficient VLA Inference](https://mlsanity.com/paper/2601.11250/) (2026)
- [ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation](https://mlsanity.com/paper/2603.13788/) (2026)
- [TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning](https://mlsanity.com/paper/2605.12236/) (2026)
- [KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learning](https://mlsanity.com/paper/2601.14232/) (2026)
- [GORDON: Graph-based Object-centric Rewards for Decomposition of Long-Horizon Manipulation](https://mlsanity.com/paper/2608.03753/) (2026)
- [ObstaDiff: Generalizable Diffusion Policy Learning via Obstacle-aware Representations](https://mlsanity.com/paper/2609.10918/) (2026)

## Abstract (arXiv)

Visuomotor policies trained on human expert demonstrations have recently shown strong performance across a wide range of robotic manipulation tasks. However, these policies remain highly sensitive to domain shifts stemming from background or robot embodiment changes, which limits their generalization capabilities. In this paper, we present ARRO, a novel visual representation that leverages zero-shot open-vocabulary segmentation and object detection models to efficiently mask out task-irrelevant regions of the scene in real time without requiring additional training, modeling of the setup, or camera calibration. By filtering visual distractors and overlaying virtual guides during both training and inference, ARRO improves robustness to scene variations and reduces the need for additional data collection. We extensively evaluate ARRO with Diffusion Policy on a range of tabletop manipulation tasks in both simulation and real-world environments, and further demonstrate its compatibility and effectiveness with generalist robot policies, such as Octo, OpenVLA and Pi Zero. Across all settings in our evaluation, ARRO yields consistent performance gains, allows for selective masking to choose between different objects, and shows robustness even to challenging segmentation conditions. Videos showcasing our results are available at: https://augmented-reality-for-robots.github.io/