# EgoOops: A Dataset for Mistake Action Detection from Egocentric Videos referring to Procedural Texts

arXiv: https://arxiv.org/abs/2410.05343 · HTML: https://mlsanity.com/paper/2410.05343/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World](https://mlsanity.com/paper/2309.17024/) (2023)
- [EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone](https://mlsanity.com/paper/2307.05463/) (2023)
- [Actor and Observer: Joint Modeling of First and Third-Person Videos](https://mlsanity.com/paper/1804.09627/) (2018)
- [My View is the Best View: Procedure Learning from Egocentric Videos](https://mlsanity.com/paper/2207.10883/) (2022)
- [CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities](https://mlsanity.com/paper/2312.14556/) (2023)
- [IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting](https://mlsanity.com/paper/2310.17323/) (2023)
- [Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers](https://mlsanity.com/paper/2108.11996/) (2021)
- [PREGO: online mistake detection in PRocedural EGOcentric videos](https://mlsanity.com/paper/2404.01933/) (2024)
- [StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos](https://mlsanity.com/paper/2304.13265/) (2023)
- [MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain](https://mlsanity.com/paper/2209.08691/) (2022)
- [Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos](https://mlsanity.com/paper/2406.01486/) (2024)

## Built on by (most cited first, 5 of 5)

- [EgoLife: Towards Egocentric Life Assistant](https://mlsanity.com/paper/2503.03803/) (2025)
- [Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges](https://mlsanity.com/paper/2511.13261/) (2025)
- [MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos](https://mlsanity.com/paper/2603.14252/) (2026)
- [How to Correctly Make Mistakes: A Framework for Constructing and Benchmarking Mistake Aware Egocentric Procedural Videos](https://mlsanity.com/paper/2604.15134/) (2026)
- [EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent](https://mlsanity.com/paper/2607.13792/) (2026)

## Abstract (arXiv)

Mistake action detection is crucial for developing intelligent archives that detect workers' errors and provide feedback. Existing studies have focused on visually apparent mistakes in free-style activities, resulting in video-only approaches to mistake detection. However, in text-following activities, models cannot determine the correctness of some actions without referring to the texts. Additionally, current mistake datasets rarely use procedural texts for video recording except for cooking. To fill these gaps, this paper proposes the EgoOops dataset, where egocentric videos record erroneous activities when following procedural texts across diverse domains. It features three types of annotations: video-text alignment, mistake labels, and descriptions for mistakes. We also propose a mistake detection approach, combining video-text alignment and mistake label classification to leverage the texts. Our experimental results show that incorporating procedural texts is essential for mistake detection. Data is available through https://y-haneji.github.io/EgoOops-project-page/.