# Using Visual Anomaly Detection for Task Execution Monitoring

arXiv: https://arxiv.org/abs/2107.14206 · HTML: https://mlsanity.com/paper/2107.14206/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 29 Jul 2021.

## Builds on (most cited first, 15 of 15)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [RoboNet: Large-Scale Multi-Robot Learning](https://mlsanity.com/paper/1910.11215/) (2019)
- [Anomaly Detection in Video Sequence with Appearance-Motion Correspondence](https://mlsanity.com/paper/1908.06351/) (2019)
- [Street Scene: A new dataset and evaluation protocol for video anomaly detection](https://mlsanity.com/paper/1902.05872/) (2019)
- [A Survey of Single-Scene Video Anomaly Detection](https://mlsanity.com/paper/2004.05993/) (2020)
- [Time-Agnostic Prediction: Predicting Predictable Video Frames](https://mlsanity.com/paper/1808.07784/) (2018)
- [Safe Robot Navigation via Multi-Modal Anomaly Detection](https://mlsanity.com/paper/2001.07934/) (2020)
- [Future Frame Prediction Using Convolutional VRNN for Anomaly Detection](https://mlsanity.com/paper/1909.02168/) (2019)
- [Towards Anomaly Detection in Dashcam Videos](https://mlsanity.com/paper/2004.05261/) (2020)
- [FINO-Net: A Deep Multimodal Sensor Fusion Framework for Manipulation Failure Detection](https://mlsanity.com/paper/2011.05817/) (2020)
- [Deep execution monitor for robot assistive tasks](https://mlsanity.com/paper/1902.02877/) (2019)

## Built on by (most cited first, 7 of 7)

- [Unsupervised Anomaly Detection from Time-of-Flight Depth Images](https://mlsanity.com/paper/2203.01052/) (2022)
- [EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents](https://mlsanity.com/paper/2609.01281/) (2026)
- [Locally Adaptive One-Class Classifier Fusion with Dynamic ℓp-Norm Constraints for Robust Anomaly Detection](https://mlsanity.com/paper/2411.06406/) (2024)
- [Enhancing Video-Based Robot Failure Detection Using Task Knowledge](https://mlsanity.com/paper/2508.18705/) (2025)
- [ViSTR-GP: Online Cyberattack Detection via Vision-to-State Tensor Regression and Gaussian Processes in Automated Robotic Operations](https://mlsanity.com/paper/2509.10948/) (2025)
- [Visual Anomaly Detection for Reliable Robotic Implantation of Flexible Microelectrode Array](https://mlsanity.com/paper/2510.09071/) (2025)
- [RoboVAD: A Large Cross-Domain Evaluation Benchmark for Anomaly Detection in Robotic Arm Manipulation Videos](https://mlsanity.com/paper/2609.17843/) (2026)

## Abstract (arXiv)

Execution monitoring is essential for robots to detect and respond to failures. Since it is impossible to enumerate all failures for a given task, we learn from successful executions of the task to detect visual anomalies during runtime. Our method learns to predict the motions that occur during the nominal execution of a task, including camera and robot body motion. A probabilistic U-Net architecture is used to learn to predict optical flow, and the robot's kinematics and 3D model are used to model camera and body motion. The errors between the observed and predicted motion are used to calculate an anomaly score. We evaluate our method on a dataset of a robot placing a book on a shelf, which includes anomalies such as falling books, camera occlusions, and robot disturbances. We find that modeling camera and body motion, in addition to the learning-based optical flow prediction, results in an improvement of the area under the receiver operating characteristic curve from 0.752 to 0.804, and the area under the precision-recall curve from 0.467 to 0.549.