# TRAFA: Anticipating User Actions to Reduce Errors in Procedural Tasks with Predictive Feedback

arXiv: https://arxiv.org/abs/2605.24526 · HTML: https://mlsanity.com/paper/2605.24526/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 49 references that are in the map.
- First posted on arXiv 23 May 2026.

## Builds on (most cited first, 7 of 7)

- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling](https://mlsanity.com/paper/2410.16668/) (2024)
- [PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch](https://mlsanity.com/paper/2407.16785/) (2024)
- [Enhancing Context-Aware Human Motion Prediction for Efficient Robot Handovers](https://mlsanity.com/paper/2503.00576/) (2025)

## Abstract (arXiv)

Interactive assistance systems typically provide feedback after an action has been completed, supporting error recovery but not preventing the error itself. We present TRAFA, a real-time predictive feedback system for procedural tasks that intervenes before errors are committed. TRAFA operationalizes predictive feedback through a Track-Forecast-Act framework that tracks hand and object state, forecasts user motion conditioned on scene context, and triggers feedback when a predicted action is likely to violate task constraints. We instantiate this pipeline in a sequential assembly setting and evaluate it through both technical benchmarking and a controlled user study against conventional reactive feedback. Our results show that predictive feedback improves task accuracy and efficiency while maintaining a comparable number of feedback events. These findings position feedback timing as a key dimension in system design and show how real-time anticipation can be integrated into interactive systems to prevent errors before they occur.