# Automated capture and delivery of assistive task guidance with an eyewear computer: The GlaciAR system

arXiv: https://arxiv.org/abs/1701.02586 · HTML: https://mlsanity.com/paper/1701.02586/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2016.

## Built on by (most cited first, 3 of 3)

- [Proactive Assistant Dialogue Generation from Streaming Egocentric Videos](https://mlsanity.com/paper/2506.05904/) (2025)
- [Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?](https://mlsanity.com/paper/2311.00738/) (2023)
- [The Object at Hand: Automated Editing for Mixed Reality Video Guidance from Hand-Object Interactions](https://mlsanity.com/paper/2109.14744/) (2021)

## Abstract (arXiv)

In this paper we describe and evaluate a mixed reality system that aims to augment users in task guidance applications by combining automated and unsupervised information collection with minimally invasive video guides. The result is a self-contained system that we call GlaciAR (Glass-enabled Contextual Interactions for Augmented Reality), that operates by extracting contextual interactions from observing users performing actions. GlaciAR is able to i) automatically determine moments of relevance based on a head motion attention model, ii) automatically produce video guidance information, iii) trigger these video guides based on an object detection method, iv) learn without supervision from observing multiple users and v) operate fully on-board a current eyewear computer (Google Glass). We describe the components of GlaciAR together with evaluations on how users are able to use the system to achieve three tasks. We see this work as a first step toward the development of systems that aim to scale up the notoriously difficult authoring problem in guidance systems and where people's natural abilities are enhanced via minimally invasive visual guidance.