mlsanityData to 7 Oct 2026

CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities

arXiv 2312.14556 ↗First posted 22 Dec 2023cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

Following step-by-step procedures is an essential component of various activities carried out by individuals in their daily lives. These procedures serve as a guiding framework that helps to achieve goals efficiently, whether it is assembling furniture or preparing a recipe. However, the complexity and duration of procedural activities inherently increase the likelihood of making errors. Understanding such procedural activities from a sequence of frames is a challenging task that demands an accurate interpretation of visual information and the ability to reason about the structure of the activity. To this end, we collect a new egocentric 4D dataset, CaptainCook4D, comprising 384 recordings (94.5 hours) of people performing recipes in real kitchen environments. This dataset consists of two distinct types of activity: one in which participants adhere to the provided recipe instructions and another in which they deviate and induce errors. We provide 5.3K step annotations and 10K fine-grained action annotations and benchmark the dataset for the following tasks: supervised error recognition, multistep localization, and procedure learning

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 87 of its 120 references that are in the map. First posted on arXiv 22 Dec 2023.

Builds on

87 of 120 references.

References · 87

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2018SlowFast Networks for Video Recognitioncited by 1,862
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  12. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
38 more
  1. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  2. 2020Rescaling Egocentric Visioncited by 529
  3. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  4. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  5. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  6. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  7. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  8. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  9. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  10. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  11. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  12. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  13. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  14. 2021Anticipative Video Transformercited by 197
  15. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  16. 2017Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognitioncited by 178
  17. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  18. 2019Temporal Cycle-Consistency Learningcited by 151
  19. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  20. 2021Self-Supervised Predictive Convolutional Attentive Block for Anomaly Detectioncited by 119
  21. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  22. 2015Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videoscited by 93
  23. 2019Procedure Planning in Instructional Videoscited by 92
  24. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  25. 2022Temporal Alignment Networks for Long-term Videocited by 84
  26. 2018Localizing Moments in Video with Temporal Languagecited by 83
  27. 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
  28. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  29. 2020The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domaincited by 79
  30. 2016Connectionist Temporal Modeling for Weakly Supervised Action Labelingcited by 70
  31. 2020In the Eye of the Beholder: Gaze and Actions in First Person Videocited by 68
  32. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  33. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  34. 2016Online Real-time Multiple Spatiotemporal Action Localisation and Predictioncited by 57
  35. 2022MAViL: Masked Audio-Video Learnerscited by 56
  36. 2022Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translationcited by 55
  37. 2019Weakly Supervised Energy-Based Learning for Action Segmentationcited by 53
  38. 2021Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learningcited by 53

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 18; 2026: 22.

47 papers.

Sort:

By area:

  1. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  2. 2024Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videoscited by 24
  3. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 22
  4. 2025Modeling Multiple Normal Action Representations for Error Detection in Procedural Taskscited by 12
  5. 2025Vid2Coach: Transforming How-To Videos into Task Assistantscited by 12
  6. 2025OpenEgo: A Large-Scale Multimodal Egocentric Dataset for Dexterous Manipulationcited by 12
  7. 2024TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videoscited by 10
  8. 2024ProMQA: Question Answering Dataset for Multimodal Procedural Activity Understandingcited by 8
  9. 2026IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assemblycited by 8
  10. 2025EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real Worldcited by 7
  11. 2024Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipationcited by 6
  12. 2024EgoOops: A Dataset for Mistake Action Detection from Egocentric Videos referring to Procedural Textscited by 5
35 more
  1. 2025ProMQA-Assembly: Multimodal Procedural QA Dataset on Assemblycited by 5
  2. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  3. 2025Procedural Mistake Detection via Action Effect Modelingcited by 5
  4. 2024Transparent and Coherent Procedural Mistake Detectioncited by 4
  5. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  6. 2025PhysLab: A Benchmark Dataset for Multi-Granularity Visual Parsing of Physics Experimentscited by 3
  7. 2026Can AI Understand the Language of Origami?cited by 3
  8. 2026OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistantscited by 3
  9. 2024"Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaborationcited by 2
  10. 2025Task Graph Maximum Likelihood Estimation for Procedural Activity Understanding in Egocentric Videoscited by 2
  11. 2025Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?cited by 2
  12. 2026ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarioscited by 2
  13. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 2
  14. 2026The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detectioncited by 2
  15. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 2
  16. 2026EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agentcited by 2
  17. 2025Position: Olfaction Standardization is Essential for the Advancement of Embodied Artificial Intelligencecited by 1
  18. 2025TAMA: Tool-Augmented Multimodal Agent for Procedural Activity Understandingcited by 1
  19. 2025Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videoscited by 1
  20. 2025ProcObject-10K: Benchmarking Object-Centric Procedural Understanding in Instructional Videoscited by 1
  21. 2026MistExit: Learning to Exit for Early Mistake Detection in Procedural Videoscited by 1
  22. 2026GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Videocited by 1
  23. 2026Post-Training VLMs for Video Mistake Detectioncited by 1
  24. 2026PROVIA: Procedure State Tracking for Online Mistake Detection in Egocentric Videoscited by 1
  25. 2025I2G: Generating Instructional Illustrations via Text-Conditioned Diffusioncited by 0
  26. 2025Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challengescited by 0
  27. 2026From Instructions to Assistance: a Dataset Aligning Instruction Manuals with Assembly Videos for Evaluating Multimodal LLMscited by 0
  28. 2026Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignmentcited by 0
  29. 2026RECIPE: Procedural Planning via Grounding in Instructional Videocited by 0
  30. 2026Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?cited by 0
  31. 2026ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loopcited by 0
  32. 2026EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AIcited by 0
  33. 2026VidParse: Online Parsing of Egocentric Procedures Like a Procited by 0
  34. 2026Hand-Aware Transition Modeling for Bimanual Procedural Anomaly Detectioncited by 0
  35. 2026Algebraic Consistency Alone Does Not Certify Temporal Structure in Latent Action Modelscited by 0