Task Graph Maximum Likelihood Estimation for Procedural Activity Understanding in Egocentric Videos
Not on the map. Why?
Abstract (arXiv)
We introduce a gradient-based approach for learning task graphs from procedural activities, improving over hand-crafted methods. Our method directly optimizes edge weights via maximum likelihood, enabling integration into neural architectures. We validate our approach on CaptainCook4D, EgoPER, and EgoProceL, achieving +14.5%, +10.2%, and +13.6% F1-score improvements. Our feature-based approach for predicting task graphs from textual/video embeddings demonstrates emerging video understanding abilities. We also achieved top performance on the procedure understanding benchmark on Ego-Exo4D and significantly improved online mistake detection (+19.8% on Assembly101-O, +6.4% on EPIC-Tent-O). Code: https://github.com/fpv-iplab/Differentiable-Task-Graph-Learning.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 29 of its 51 references that are in the map. First posted on arXiv 25 Feb 2025.
Builds on
29 of 51 references.
References · 29
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
- 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
- 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
- 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
- 2019Cross-task weakly supervised learning from instructional videoscited by 199
- 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
- 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
- 2023An Outlook into the Future of Egocentric Visioncited by 84
17 more
- 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
- 2020The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domaincited by 79
- 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
- 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
- 2023Learning Procedure-aware Video Representation from Instructional Videos and Their Narrationscited by 46
- 2023StepFormer: Self-supervised Step Discovery and Localization in Instructional Videoscited by 37
- 2024PREGO: online mistake detection in PRocedural EGOcentric videoscited by 36
- 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
- 2023Learning and Verification of Task Structure in Instructional Videoscited by 26
- 2020Meta Reinforcement Learning with Autonomous Inference of Subtask Dependenciescited by 25
- 2024Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videoscited by 24
- 2023Every Mistake Counts in Assemblycited by 18
- 2020Egocentric Object Manipulation Graphscited by 17
- 2024Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videoscited by 15
- 2023Multimodal Subtask Graph Generation from Instructional Videoscited by 13
- 2023United We Stand, Divided We Fall: UnityGraph for Unsupervised Procedure Learning from Videoscited by 13
- 2022Graph2Vid: Flow graph to Video Grounding for Weakly-supervised Multi-Step Localizationcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.