# fARfetch: Enabling Collocated AR-HRC in Large Visually Diverse Environments with VLM-Driven AR Content Adaptation

arXiv: https://arxiv.org/abs/2606.25162 · HTML: https://mlsanity.com/paper/2606.25162/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [The Marathon 2: A Navigation System](https://mlsanity.com/paper/2003.00368/) (2020)
- [Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces](https://mlsanity.com/paper/2203.03254/) (2022)
- [ViDDAR: Vision Language Model-Based Task-Detrimental Content Detection for Augmented Reality](https://mlsanity.com/paper/2501.12553/) (2025)
- [A 3D Mixed Reality Interface for Human-Robot Teaming](https://mlsanity.com/paper/2310.02392/) (2023)
- [ARROCH: Augmented Reality for Robots Collaborating with a Human](https://mlsanity.com/paper/2109.10400/) (2021)
- [DroneARchery: Human-Drone Interaction through Augmented Reality with Haptic Feedback and Multi-UAV Collision Avoidance Driven by Deep Reinforcement Learning](https://mlsanity.com/paper/2210.07730/) (2022)
- [RICO-MR: An Open-Source Architecture for Robot Intent Communication through Mixed Reality](https://mlsanity.com/paper/2309.04765/) (2023)
- [SafeSpect: Safety-First Augmented Reality Heads-up Display for Drone Inspections](https://mlsanity.com/paper/2504.16533/) (2025)
- [AdjustAR: AI-Driven In-Situ Adjustment of Site-Specific Augmented Reality Content](https://mlsanity.com/paper/2508.06826/) (2025)

## Abstract (arXiv)

Augmented Reality (AR) can improve collocated human-robot collaboration by making robot state and intent visible and enabling intuitive control, yet large, visually diverse environments like the outdoors challenge both interaction and content legibility, especially at long distances and beyond visual line of sight. We present fARfetch, an AR-HRC system that integrates (i) shared semantic environment mapping across an AR headset and robot that visualizes detected landmarks in AR to support landmark-grounded go-to commands, (ii) a context-aware world-in-miniature representation of the shared environment for fine-grained path authoring, and (iii) vision-language-model driven AR view management that jointly adapts virtual content color, size, and orientation to maintain legibility in large visually diverse environments. We implement fARfetch with a Meta Quest 3 headset and Unitree Go2 quadruped robot, and conduct a within-subjects user study (N=13) on a real-world large-scale (30.5m) outdoor inspection task. fARfetch yielded significantly faster completion times than a non-AR baseline (66%) and significantly lower workload in mental demand (-43%), temporal demand (-34%), and frustration (-66%). A custom legibility survey indicated fARfetch effectively maintained virtual content legibility in the large outdoor environment.