# IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting

arXiv: https://arxiv.org/abs/2310.17323 · HTML: https://mlsanity.com/paper/2310.17323/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects](https://mlsanity.com/paper/1701.05498/) (2017)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose](https://mlsanity.com/paper/2007.00394/) (2020)
- [Learning To Recognize Procedural Activities with Distant Supervision](https://mlsanity.com/paper/2201.10990/) (2022)
- [My View is the Best View: Procedure Learning from Egocentric Videos](https://mlsanity.com/paper/2207.10883/) (2022)
- [The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain](https://mlsanity.com/paper/2010.05654/) (2020)
- [Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation](https://mlsanity.com/paper/1602.02995/) (2016)
- [DeepPhase: Surgical Phase Recognition in CATARACTS Videos](https://mlsanity.com/paper/1807.10565/) (2018)
- [Unity Perception: Generate Synthetic Data for Computer Vision](https://mlsanity.com/paper/2107.04259/) (2021)
- [MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain](https://mlsanity.com/paper/2209.08691/) (2022)
- [HoloLens 2 Research Mode as a Tool for Computer Vision Research](https://mlsanity.com/paper/2008.11239/) (2020)
- [Egocentric Video Task Translation](https://mlsanity.com/paper/2212.06301/) (2022)
- [ATTACH Dataset: Annotated Two-Handed Assembly Actions for Human Action Understanding](https://mlsanity.com/paper/2304.08210/) (2023)
- [IKEA Object State Dataset: A 6DoF object pose estimation dataset and benchmark for multi-state assembly objects](https://mlsanity.com/paper/2111.08614/) (2021)
- [Human Action Forecasting by Learning Task Grammars](https://mlsanity.com/paper/1709.06391/) (2017)

## Built on by (most cited first, 30 of 48)

- [HD-EPIC: A Highly-Detailed Egocentric Video Dataset](https://mlsanity.com/paper/2502.04144/) (2025)
- [EgoLife: Towards Egocentric Life Assistant](https://mlsanity.com/paper/2503.03803/) (2025)
- [CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities](https://mlsanity.com/paper/2312.14556/) (2023)
- [PREGO: online mistake detection in PRocedural EGOcentric videos](https://mlsanity.com/paper/2404.01933/) (2024)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activities](https://mlsanity.com/paper/2406.08379/) (2024)
- [Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition](https://mlsanity.com/paper/2405.19917/) (2024)
- [Modeling Multiple Normal Action Representations for Error Detection in Procedural Tasks](https://mlsanity.com/paper/2503.22405/) (2025)
- [PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch](https://mlsanity.com/paper/2407.16785/) (2024)
- [TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videos](https://mlsanity.com/paper/2411.02570/) (2024)
- [ProMQA: Question Answering Dataset for Multimodal Procedural Activity Understanding](https://mlsanity.com/paper/2410.22211/) (2024)
- [IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assembly](https://mlsanity.com/paper/2604.10409/) (2026)
- [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024)
- [Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance](https://mlsanity.com/paper/2606.04970/) (2026)
- [ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose Estimation](https://mlsanity.com/paper/2403.16400/) (2024)
- [Procedural Mistake Detection via Action Effect Modeling](https://mlsanity.com/paper/2512.03474/) (2025)
- [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026)
- [TSAK: Two-Stage Semantic-Aware Knowledge Distillation for Efficient Wearable Modality and Model Optimization in Manufacturing Lines](https://mlsanity.com/paper/2408.14146/) (2024)
- [EgoOops: A Dataset for Mistake Action Detection from Egocentric Videos referring to Procedural Texts](https://mlsanity.com/paper/2410.05343/) (2024)
- [ProMQA-Assembly: Multimodal Procedural QA Dataset on Assembly](https://mlsanity.com/paper/2509.02949/) (2025)
- [Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges](https://mlsanity.com/paper/2511.13261/) (2025)
- [Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Context](https://mlsanity.com/paper/2410.21275/) (2024)
- [PhysLab: A Benchmark Dataset for Multi-Granularity Visual Parsing of Physics Experiments](https://mlsanity.com/paper/2506.06631/) (2025)
- [Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance](https://mlsanity.com/paper/2512.13238/) (2025)
- [Supervised Representation Learning towards Generalizable Assembly State Recognition](https://mlsanity.com/paper/2408.11700/) (2024)
- [Learning to Recognize Correctly Completed Procedure Steps in Egocentric Assembly Videos through Spatio-Temporal Modeling](https://mlsanity.com/paper/2510.12385/) (2025)
- [Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmark](https://mlsanity.com/paper/2601.00092/) (2025)
- ["Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration](https://mlsanity.com/paper/2409.10525/) (2024)
- [OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments](https://mlsanity.com/paper/2603.02390/) (2026)
- [ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios](https://mlsanity.com/paper/2603.09741/) (2026)

## Abstract (arXiv)

Although action recognition for procedural tasks has received notable attention, it has a fundamental flaw in that no measure of success for actions is provided. This limits the applicability of such systems especially within the industrial domain, since the outcome of procedural actions is often significantly more important than the mere execution. To address this limitation, we define the novel task of procedure step recognition (PSR), focusing on recognizing the correct completion and order of procedural steps. Alongside the new task, we also present the multi-modal IndustReal dataset. Unlike currently available datasets, IndustReal contains procedural errors (such as omissions) as well as execution errors. A significant part of these errors are exclusively present in the validation and test sets, making IndustReal suitable to evaluate robustness of algorithms to new, unseen mistakes. Additionally, to encourage reproducibility and allow for scalable approaches trained on synthetic data, the 3D models of all parts are publicly available. Annotations and benchmark performance are provided for action recognition and assembly state detection, as well as the new PSR task. IndustReal, along with the code and model weights, is available at: https://github.com/TimSchoonbeek/IndustReal .