mlsanityData to 7 Oct 2026

PREGO: online mistake detection in PRocedural EGOcentric videos

arXiv 2404.01933 ↗First posted 2 Apr 2024cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

Promptly identifying procedural errors from egocentric videos in an online setting is highly challenging and valuable for detecting mistakes as soon as they happen. This capability has a wide range of applications across various fields, such as manufacturing and healthcare. The nature of procedural mistakes is open-set since novel types of failures might occur, which calls for one-class classifiers trained on correctly executed procedures. However, no technique can currently detect open-set procedural mistakes online. We propose PREGO, the first online one-class classification model for mistake detection in PRocedural EGOcentric videos. PREGO is based on an online action recognition component to model the current action, and a symbolic reasoning module to predict the next actions. Mistake detection is performed by comparing the recognized current action with the expected future one. We evaluate PREGO on two procedural egocentric video datasets, Assembly101 and Epic-tent, which we adapt for online benchmarking of procedural mistake detection to establish suitable benchmarks, thus defining the Assembly101-O and Epic-tent-O datasets, respectively.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

33 of 43 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  6. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  7. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  8. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  9. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  10. 2023Larger language models do in-context learning differentlycited by 386
  11. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  12. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
21 more
  1. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  2. 2023Large Language Models as General Pattern Machinescited by 196
  3. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  4. 2022Generative Cooperative Learning for Unsupervised Video Anomaly Detectioncited by 102
  5. 2020The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Posecited by 95
  6. 2023An Outlook into the Future of Egocentric Visioncited by 84
  7. 2021OadTR: Online Action Detection with Transformerscited by 83
  8. 2020The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domaincited by 79
  9. 2023Learning Procedure-aware Video Representation from Instructional Videos and Their Narrationscited by 46
  10. 2023IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Settingcited by 46
  11. 2023Multimodal Motion Conditioned Diffusion Model for Skeleton-based Video Anomaly Detectioncited by 42
  12. 2022Plansformer: Generating Symbolic Plans using Transformerscited by 41
  13. 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
  14. 2021Self-Regulated Learning for Egocentric Video Activity Anticipationcited by 32
  15. 2021GPT3-to-plan: Extracting plans from text using GPT-3cited by 26
  16. 2023Language Models can be Logical Solverscited by 24
  17. 2023Every Mistake Counts in Assemblycited by 18
  18. 2021SVIP: Sequence VerIfication for Procedures in Videoscited by 16
  19. 2023STEPs: Self-Supervised Key Step Extraction and Localization from Unlabeled Procedural Videoscited by 13
  20. 2023ENIGMA-51: Towards a Fine-Grained Understanding of Human-Object Interactions in Industrial Scenarioscited by 11
  21. 2023GePSAn: Generative Procedure Step Anticipation in Cooking Videoscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 17; 2026: 11.

36 papers.

Sort:

By area:

  1. 2024Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videoscited by 24
  2. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  3. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 18
  4. 2024Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activitiescited by 15
  5. 2025Modeling Multiple Normal Action Representations for Error Detection in Procedural Taskscited by 12
  6. 2024TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videoscited by 10
  7. 2024EgoOops: A Dataset for Mistake Action Detection from Egocentric Videos referring to Procedural Textscited by 5
  8. 2025ProMQA-Assembly: Multimodal Procedural QA Dataset on Assemblycited by 5
  9. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  10. 2025Procedural Mistake Detection via Action Effect Modelingcited by 5
  11. 2024Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Contextcited by 4
  12. 2024Transparent and Coherent Procedural Mistake Detectioncited by 4
24 more
  1. 2025YETI (YET to Intervene) Proactive Interventions by Multimodal AI Agents in Augmented Reality Taskscited by 4
  2. 2025EvoStruggle: A Dataset Capturing the Evolution of Struggle across Activities and Skill Levelscited by 3
  3. 2025Learning to Recognize Correctly Completed Procedure Steps in Egocentric Assembly Videos through Spatio-Temporal Modelingcited by 3
  4. 2025Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistancecited by 3
  5. 2025Task Graph Maximum Likelihood Estimation for Procedural Activity Understanding in Egocentric Videoscited by 2
  6. 2025HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamicscited by 2
  7. 2026Pro2Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural Taskscited by 2
  8. 2026The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detectioncited by 2
  9. 2024ACE: Action Concept Enhancement of Video-Language Models in Procedural Videoscited by 1
  10. 2025Technical Report for Egocentric Mistake Detection for the HoloAssist Challengecited by 1
  11. 2025Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videoscited by 1
  12. 2026MistExit: Learning to Exit for Early Mistake Detection in Procedural Videoscited by 1
  13. 2026How to Correctly Make Mistakes: A Framework for Constructing and Benchmarking Mistake Aware Egocentric Procedural Videoscited by 1
  14. 2026ESTANet: Efficient Online Error Detection in Procedural Videos via Prediction Inconsistencycited by 1
  15. 2026Post-Training VLMs for Video Mistake Detectioncited by 1
  16. 2026PROVIA: Procedure State Tracking for Online Mistake Detection in Egocentric Videoscited by 1
  17. 2024Find the Assembly Mistakes: Error Segmentation for Industrial Applicationscited by 0
  18. 2025EgoInstruct: An Egocentric Video Dataset of Face-to-face Instructional Interactions with Multi-modal LLM Benchmarkingcited by 0
  19. 2025Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challengescited by 0
  20. 2025From Detection to Anticipation: Online Understanding of Struggles across Various Tasks and Activitiescited by 0
  21. 2026SAVA-X: Ego-to-Exo Imitation Error Detection via Scene-Adaptive View Alignment and Bidirectional Cross View Fusioncited by 0
  22. 2026EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AIcited by 0
  23. 2026INSPECT: Learning Robot View Selection from Assistant Usecited by 0
  24. 2026Hand-Aware Transition Modeling for Bimanual Procedural Anomaly Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RegulAR: Graph-Grounded Error Recognition and Assistance for Procedural Tasks in AR
  2. 2026EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next
  3. 2024ALGO: Object-Grounded Visual Commonsense Reasoning for Open-World Egocentric Action Recognition
  4. 2025Dual-Stage Reweighted MoE for Long-Tailed Egocentric Mistake Detection
  5. 2024EAGLE: Egocentric AGgregated Language-video Engine
  6. 2023Discovering Novel Actions from Open World Egocentric Videos with Object-Grounded Visual Commonsense Reasoning
  7. 2026EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory
  8. 2026SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos