# Enhancing Video-Based Robot Failure Detection Using Task Knowledge

arXiv: https://arxiv.org/abs/2508.18705 · HTML: https://mlsanity.com/paper/2508.18705/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 26 Aug 2025.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder](https://mlsanity.com/paper/1711.00614/) (2017)
- [Vision-Language Models as Success Detectors](https://mlsanity.com/paper/2303.07280/) (2023)
- [AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation](https://mlsanity.com/paper/2410.00371/) (2024)
- [Manipulate-Anything: Automating Real-World Robots using Vision-Language Models](https://mlsanity.com/paper/2406.18915/) (2024)
- [Oops! Predicting Unintentional Action in Video](https://mlsanity.com/paper/1911.11206/) (2019)
- [SMART Frame Selection for Action Recognition](https://mlsanity.com/paper/2012.10671/) (2020)
- [Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress](https://mlsanity.com/paper/2410.04640/) (2024)
- [MGSampler: An Explainable Sampling Strategy for Video Action Recognition](https://mlsanity.com/paper/2104.09952/) (2021)
- [Learning Representational Invariances for Data-Efficient Action Recognition](https://mlsanity.com/paper/2103.16565/) (2021)
- [MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/2006.09220/) (2020)
- [ARMBench: An Object-centric Benchmark Dataset for Robotic Manipulation](https://mlsanity.com/paper/2303.16382/) (2023)
- [Proactive Anomaly Detection for Robot Navigation with Multi-Sensor Fusion](https://mlsanity.com/paper/2204.01146/) (2022)
- [FINO-Net: A Deep Multimodal Sensor Fusion Framework for Manipulation Failure Detection](https://mlsanity.com/paper/2011.05817/) (2020)
- [Using Visual Anomaly Detection for Task Execution Monitoring](https://mlsanity.com/paper/2107.14206/) (2021)
- [PMI Sampler: Patch Similarity Guided Frame Selection for Aerial Action Recognition](https://mlsanity.com/paper/2304.06866/) (2023)
- [ConditionNET: Learning Preconditions and Effects for Execution Monitoring](https://mlsanity.com/paper/2502.01167/) (2025)
- [Grasp State Assessment of Deformable Objects Using Visual-Tactile Fusion Perception](https://mlsanity.com/paper/2006.12729/) (2020)
- [A Multimodal Handover Failure Detection Dataset and Baselines](https://mlsanity.com/paper/2402.18319/) (2024)
- [FrAUG: A Frame Rate Based Data Augmentation Method for Depression Detection from Speech Signals](https://mlsanity.com/paper/2202.05912/) (2022)
- [Variable frame rate-based data augmentation to handle speaking-style variability for automatic speaker verification](https://mlsanity.com/paper/2008.03616/) (2020)
- [Sensing Anomalies as Potential Hazards: Datasets and Benchmarks](https://mlsanity.com/paper/2110.14706/) (2021)
- [Scalable Frame Sampling for Video Classification: A Semi-Optimal Policy Approach with Reduced Search Space](https://mlsanity.com/paper/2409.05260/) (2024)

## Abstract (arXiv)

Robust robotic task execution hinges on the reliable detection of execution failures in order to trigger safe operation modes, recovery strategies, or task replanning. However, many failure detection methods struggle to provide meaningful performance when applied to a variety of real-world scenarios. In this paper, we propose a video-based failure detection approach that uses spatio-temporal knowledge in the form of the actions the robot performs and task-relevant objects within the field of view. Both pieces of information are available in most robotic scenarios and can thus be readily obtained. We demonstrate the effectiveness of our approach on three datasets that we amend, in part, with additional annotations of the aforementioned task-relevant knowledge. In light of the results, we also propose a data augmentation method that improves performance by applying variable frame rates to different parts of the video. We observe an improvement from 77.9 to 80.0 in F1 score on the ARMBench dataset without additional computational expense and an additional increase to 81.4 with test-time augmentation. The results emphasize the importance of spatio-temporal information during failure detection and suggest further investigation of suitable heuristics in future implementations. Code and annotations are available.