mlsanityData to 14 Oct 2026

Procedure Planning in Instructional Videos

arXiv 1907.01172 ↗First posted 2 Jul 2019cs.CVcited by 92

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the problem of procedure planning in instructional videos, which can be seen as a step towards enabling autonomous agents to plan for complex tasks in everyday settings such as cooking. Given the current visual observation of the world and a visual goal, we ask the question "What actions need to be taken in order to achieve the goal?". The key technical challenge is to learn structured and plannable state and action spaces directly from unstructured videos. We address this challenge by proposing Dual Dynamics Networks (DDN), a framework that explicitly leverages the structured priors imposed by the conjugate relationships between states and actions in a learned plannable latent space. We evaluate our method on real-world instructional videos. Our experiments show that DDN learns plannable representations that lead to better planning performance compared to existing planning approaches and neural network policies.

Key facts. Cited by 92 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 4; 2022: 16; 2023: 25; 2024: 18; 2025: 15; 2026: 13.

92 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  2. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  3. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 115
  4. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 81
  5. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 69
  6. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  7. 2021Home Action Genome: Cooperative Compositional Action Understandingcited by 53
  8. 2021Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learningcited by 53
  9. 2022P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervisioncited by 52
  10. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 51
  11. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  12. 2023Learning Procedure-aware Video Representation from Instructional Videos and Their Narrationscited by 46
38 more
  1. 2019Imitation Learning for Human Pose Predictioncited by 44
  2. 2023Multimodal Procedural Planning via Dual Text-Image Promptingcited by 40
  3. 2022Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videoscited by 39
  4. 2023StepFormer: Self-supervised Step Discovery and Localization in Instructional Videoscited by 37
  5. 2023PDPP: Projected Diffusion for Procedure Planning in Instructional Videoscited by 36
  6. 2023Learning Object State Changes in Videos: An Open-World Perspectivecited by 36
  7. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  8. 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
  9. 2022Visual Abductive Reasoningcited by 30
  10. 2022Rethinking Learning Approaches for Long-Term Action Anticipationcited by 29
  11. 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
  12. 2023Pretrained Language Models as Visual Planners for Human Assistancecited by 28
  13. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  14. 2023Learning and Verification of Task Structure in Instructional Videoscited by 26
  15. 2024SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videoscited by 24
  16. 2023Event-Guided Procedure Planning from Instructional Videos with Text Supervisioncited by 21
  17. 2025WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planningcited by 20
  18. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 19
  19. 2023Every Mistake Counts in Assemblycited by 18
  20. 2024ExpertAF: Expert Actionable Feedback from Videocited by 17
  21. 2024Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videoscited by 15
  22. 2022Predicting the Next Action by Modeling the Abstract Goalcited by 14
  23. 2023Skip-Plan: Procedure Planning in Instructional Videos via Condensed Action Space Learningcited by 14
  24. 2022Sports Video Analysis on Large-Scale Datacited by 13
  25. 2022CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answeringcited by 13
  26. 2022Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videoscited by 12
  27. 2024Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videoscited by 12
  28. 2021Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videoscited by 11
  29. 2022Reasoning about Actions over Visual and Linguistic Modalities: A Surveycited by 10
  30. 2024Enhancing Temporal Modeling of Video LLMs via Time Gatingcited by 10
  31. 2024PlanLLM: Video Procedure Planning with Refinable Large Language Modelscited by 10
  32. 2023GePSAn: Generative Procedure Step Anticipation in Cooking Videoscited by 9
  33. 2024Language Guided Exploration for RL Agents in Text Environmentscited by 9
  34. 2024Human Action Anticipation: A Surveycited by 9
  35. 2024FIction: 4D Future Interaction Prediction from Videocited by 9
  36. 2023Action Dynamics Task Graphs for Learning Plannable Representations of Procedural Taskscited by 8
  37. 2024ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videoscited by 8
  38. 2023Masked Diffusion with Task-awareness for Procedure Planning in Instructional Videoscited by 7

50 most cited of 92 All 92 (JSON)