mlsanityData to 7 Oct 2026

United We Stand, Divided We Fall: UnityGraph for Unsupervised Procedure Learning from Videos

arXiv 2311.03550 ↗First posted 6 Nov 2023cs.CV cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Given multiple videos of the same task, procedure learning addresses identifying the key-steps and determining their order to perform the task. For this purpose, existing approaches use the signal generated from a pair of videos. This makes key-steps discovery challenging as the algorithms lack inter-videos perspective. Instead, we propose an unsupervised Graph-based Procedure Learning (GPL) framework. GPL consists of the novel UnityGraph that represents all the videos of a task as a graph to obtain both intra-video and inter-videos context. Further, to obtain similar embeddings for the same key-steps, the embeddings of UnityGraph are updated in an unsupervised manner using the Node2Vec algorithm. Finally, to identify the key-steps, we cluster the embeddings using KMeans. We test GPL on benchmark ProceL, CrossTask, and EgoProceL datasets and achieve an average improvement of 2% on third-person datasets and 3.6% on EgoProceL over the state-of-the-art.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 45 of its 73 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

45 of 73 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  4. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  5. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
  6. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  7. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  8. 2016Generating Videos with Scene Dynamicscited by 376
  9. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  10. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 248
  11. 2018Videos as Space-Time Region Graphscited by 221
  12. 2017Unsupervised Representation Learning by Sorting Sequencescited by 219
33 more
  1. 2019Graph Convolutional Networks for Temporal Action Localizationcited by 208
  2. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  3. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  4. 2019Temporal Cycle-Consistency Learningcited by 151
  5. 2018Self-Supervised Video Representation Learning with Space-Time Cubic Puzzlescited by 149
  6. 2018Graph-Based Global Reasoning Networkscited by 120
  7. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  8. 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
  9. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  10. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  11. 2018Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignmentcited by 74
  12. 2016Connectionist Temporal Modeling for Weakly Supervised Action Labelingcited by 70
  13. 2018NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learningcited by 67
  14. 2018Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstrationcited by 60
  15. 2019VideoGraph: Recognizing Minutes-Long Human Activities in Videoscited by 57
  16. 2019Weakly Supervised Energy-Based Learning for Action Segmentationcited by 53
  17. 2022P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervisioncited by 52
  18. 2014Weakly Supervised Action Labeling in Videos Under Ordering Constraintscited by 48
  19. 2019DynamoNet: Dynamic Action and Motion Networkcited by 48
  20. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 48
  21. 2020Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequencescited by 45
  22. 2018Zero-Shot Anticipation for Instructional Activitiescited by 43
  23. 2020Set-Constrained Viterbi for Set-Supervised Action Segmentationcited by 35
  24. 2022TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliencycited by 35
  25. 2015What's Cookin'? Interpreting Cooking Videos using Text, Speech and Visioncited by 31
  26. 2019Action Modifiers: Learning from Adverbs in Instructional Videoscited by 27
  27. 2015Unsupervised Semantic Parsing of Video Collectionscited by 26
  28. 2019Learning Video Representations from Correspondence Proposalscited by 22
  29. 2020Learning to Segment Actions from Observation and Narrationcited by 21
  30. 2021SVIP: Sequence VerIfication for Procedures in Videoscited by 16
  31. 2018DiscrimNet: Semi-Supervised Action Recognition from Videos using Generative Adversarial Networkscited by 14
  32. 2021MEGAN: Memory Enhanced Graph Attention Network for Space-Time Video Super-Resolutioncited by 8
  33. 2022Self-supervised Video Representation Learning with Cascade Positive Retrievalcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videoscited by 24
  2. 2025HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videoscited by 7
  3. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  4. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  5. 2025Procedure Learning via Regularized Gromov-Wasserstein Optimal Transportcited by 4
  6. 2025Task Graph Maximum Likelihood Estimation for Procedural Activity Understanding in Egocentric Videoscited by 2
  7. 2026EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agentcited by 2
  8. 2025Leveraging Procedural Knowledge and Task Hierarchies for Efficient Instructional Video Pre-trainingcited by 1
  9. 2025Stitch-a-Demo: Video Demonstrations from Multistep Descriptionscited by 1
  10. 2025REMAP: Regularized Matching and Partial Alignment of Video Embeddingscited by 1
  11. 2025Generalized Event Partonomy Inference with Structured Hierarchical Predictive Learningcited by 0
  12. 2025The N-Body Problem: Parallel Execution from Single-Person Egocentric Videocited by 0
1 more
  1. 2026P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecturecited by 0