mlsanityData to 14 Oct 2026

Video-Mined Task Graphs for Keystep Recognition in Instructional Videos

arXiv 2307.08763 ↗First posted 17 Jul 2023cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Procedural activity understanding requires perceiving human actions in terms of a broader task, where multiple keysteps are performed in sequence across a long video to reach a final goal state -- such as the steps of a recipe or a DIY fix-it task. Prior work largely treats keystep recognition in isolation of this broader structure, or else rigidly confines keysteps to align with a predefined sequential script. We propose discovering a task graph automatically from how-to videos to represent probabilistically how people tend to execute keysteps, and then leverage this graph to regularize keystep recognition in novel videos. On multiple datasets of real-world instructional videos, we show the impact: more reliable zero-shot keystep localization and improved video representation learning, exceeding the state of the art.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 70 of its 86 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

70 of 86 references.

References · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2018SlowFast Networks for Video Recognitioncited by 1,867
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  8. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  10. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  11. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  12. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
38 more
  1. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  2. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  3. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  4. 2018A Short Note about Kinetics-600cited by 333
  5. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  6. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  7. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  8. 2022Egocentric Video-Language Pretrainingcited by 232
  9. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  10. 2018Videos as Space-Time Region Graphscited by 221
  11. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  12. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  13. 2021Anticipative Video Transformercited by 200
  14. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  15. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  16. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  17. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  18. 2019Few-Shot Video Classification via Temporal Alignmentcited by 125
  19. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 120
  20. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  21. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  22. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  23. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107
  24. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  25. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  26. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  27. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  28. 2019Procedure Planning in Instructional Videoscited by 92
  29. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  30. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  31. 2022Temporal Alignment Networks for Long-term Videocited by 84
  32. 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
  33. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  34. 2020Location-aware Graph Convolutional Networks for Video Question Answeringcited by 80
  35. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  36. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 78
  37. 2018Object Level Visual Reasoning in Videoscited by 67
  38. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 18; 2026: 6.

35 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  2. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  3. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  4. 2024PREGO: online mistake detection in PRocedural EGOcentric videoscited by 39
  5. 2024Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videoscited by 26
  6. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 19
  7. 2024ExpertAF: Expert Actionable Feedback from Videocited by 17
  8. 2024Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videoscited by 15
  9. 2025Vid2Coach: Transforming How-To Videos into Task Assistantscited by 12
  10. 2024FIction: 4D Future Interaction Prediction from Videocited by 9
  11. 2024Detours for Navigating Instructional Videoscited by 7
  12. 2025VLog: Video-Language Models by Generative Retrieval of Narration Vocabularycited by 7
23 more
  1. 2025HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videoscited by 7
  2. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  3. 2025Procedural Mistake Detection via Action Effect Modelingcited by 6
  4. 2024HANDI: Hand-Centric Text-and-Image Conditioned Video Generationcited by 5
  5. 2025LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistantscited by 5
  6. 2025ProMQA-Assembly: Multimodal Procedural QA Dataset on Assemblycited by 5
  7. 2025Learning Skill-Attributes for Transferable Assessment in Videocited by 4
  8. 2025Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistancecited by 4
  9. 2026ViterbiPlanNet: Injecting Procedural Knowledge via Differentiable Viterbi for Planning in Instructional Videoscited by 4
  10. 2025Neuro Symbolic Knowledge Reasoning for Procedural Video Question Answeringcited by 3
  11. 2025Learning to Recognize Correctly Completed Procedure Steps in Egocentric Assembly Videos through Spatio-Temporal Modelingcited by 3
  12. 2025Task Graph Maximum Likelihood Estimation for Procedural Activity Understanding in Egocentric Videoscited by 2
  13. 2025CLAD: Constrained Latent Action Diffusion for Vision-Language Procedure Planningcited by 2
  14. 2026ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarioscited by 2
  15. 2026Demo2Tutorial: From Human Experience to Multimodal Software Tutorialscited by 2
  16. 2024Switch-a-View: View Selection Learned from Unlabeled In-the-wild Videoscited by 1
  17. 2025Stitch-a-Demo: Video Demonstrations from Multistep Descriptionscited by 1
  18. 2025ProcObject-10K: Benchmarking Object-Centric Procedural Understanding in Instructional Videoscited by 1
  19. 2026SportSkills: Physical Skill Learning from Sports Instructional Videoscited by 1
  20. 2026ESTANet: Efficient Online Error Detection in Procedural Videos via Prediction Inconsistencycited by 1
  21. 2025ViewBridge: Curriculum Knowledge Distillation for Activity View-Invariance Under Extreme Viewpoint Changescited by 0
  22. 2025The N-Body Problem: Parallel Execution from Single-Person Egocentric Videocited by 0
  23. 2026MoTE: Mixture of Task Experts for Multi-Task Video Understandingcited by 0