# PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch

arXiv: https://arxiv.org/abs/2407.16785 · HTML: https://mlsanity.com/paper/2407.16785/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 61 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 7 of 7)

- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World](https://mlsanity.com/paper/2309.17024/) (2023)
- [Ensembles of Deep LSTM Learners for Activity Recognition using Wearables](https://mlsanity.com/paper/1703.09370/) (2017)
- [IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting](https://mlsanity.com/paper/2310.17323/) (2023)
- [REsCUE: A framework for REal-time feedback on behavioral CUEs using multimodal anomaly detection](https://mlsanity.com/paper/1903.11485/) (2019)
- [A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operations](https://mlsanity.com/paper/2008.01513/) (2020)
- [Mindless Attractor: A False-Positive Resistant Intervention for Drawing Attention Using Auditory Perturbation](https://mlsanity.com/paper/2101.08621/) (2021)

## Built on by (most cited first, 11 of 11)

- [ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices](https://mlsanity.com/paper/2507.21378/) (2025)
- [Vid2Coach: Transforming How-To Videos into Task Assistants](https://mlsanity.com/paper/2506.00717/) (2025)
- [Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance](https://mlsanity.com/paper/2606.04970/) (2026)
- [When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Tasks](https://mlsanity.com/paper/2510.05307/) (2025)
- [Carbon and Silicon, Coexist or Compete? A Survey on Human-AI Interactions in Agent-based Modeling and Simulation](https://mlsanity.com/paper/2502.18145/) (2025)
- [Pro2Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural Tasks](https://mlsanity.com/paper/2605.04227/) (2026)
- [Everyday AR through AI-in-the-Loop](https://mlsanity.com/paper/2412.12681/) (2024)
- [Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU](https://mlsanity.com/paper/2602.15707/) (2026)
- [TransfHAR: Self-Supervised Wrist Representations for On-Demand Activity Recognition](https://mlsanity.com/paper/2608.15861/) (2026)
- [TRAFA: Anticipating User Actions to Reduce Errors in Procedural Tasks with Predictive Feedback](https://mlsanity.com/paper/2605.24526/) (2026)
- [Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks](https://mlsanity.com/paper/2609.24955/) (2026)

## Abstract (arXiv)

We routinely perform procedures (such as cooking) that include a set of atomic steps. Often, inadvertent omission or misordering of a single step can lead to serious consequences, especially for those experiencing cognitive challenges such as dementia. This paper introduces PrISM-Observer, a smartwatch-based, context-aware, real-time intervention system designed to support daily tasks by preventing errors. Unlike traditional systems that require users to seek out information, the agent observes user actions and intervenes proactively. This capability is enabled by the agent's ability to continuously update its belief in the user's behavior in real-time through multimodal sensing and forecast optimal intervention moments and methods. We first validated the steps-tracking performance of our framework through evaluations across three datasets with different complexities. Then, we implemented a real-time agent system using a smartwatch and conducted a user study in a cooking task scenario. The system generated helpful interventions, and we gained positive feedback from the participants. The general applicability of PrISM-Observer to daily tasks promises broad applications, for instance, including support for users requiring more involved interventions, such as people with dementia or post-surgical patients.