mlsanityData to 7 Oct 2026

Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videos

arXiv 2110.05697 ↗First posted 12 Oct 2021cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

This paper focuses on task recognition and action segmentation in weakly-labeled instructional videos, where only the ordered sequence of video-level actions is available during training. We propose a two-stream framework, which exploits semantic and temporal hierarchies to recognize top-level tasks in instructional videos. Further, we present a novel top-down weakly-supervised action segmentation approach, where the predicted task is used to constrain the inference of fine-grained action sequences. Experimental results on the popular Breakfast and Cooking 2 datasets show that our two-stream hierarchical task modeling significantly outperforms existing methods in top-level task recognition for all datasets and metrics. Additionally, using our task recognition framework in the proposed top-down action segmentation approach consistently improves the state of the art, while also reducing segmentation inference time by 80-90 percent.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

44 of 58 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  4. 2017Non-local Neural Networkscited by 2,290
  5. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  6. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  7. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  8. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  9. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  10. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  11. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  12. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
32 more
  1. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  2. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  3. 2019STM: SpatioTemporal and Motion Encoding for Action Recognitioncited by 142
  4. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  5. 2017Weakly Supervised Action Localization by Sparse Temporal Pooling Networkcited by 100
  6. 2019What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessmentcited by 100
  7. 2019Procedure Planning in Instructional Videoscited by 92
  8. 2018Timeception for Complex Action Recognitioncited by 89
  9. 2018W-TALC: Weakly-supervised Temporal Activity Localization and Classificationcited by 87
  10. 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
  11. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  12. 2018Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignmentcited by 74
  13. 2016Connectionist Temporal Modeling for Weakly Supervised Action Labelingcited by 70
  14. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
  15. 2018The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videoscited by 68
  16. 2018NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learningcited by 67
  17. 2015An end-to-end generative framework for video segmentation and recognitioncited by 61
  18. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  19. 2019VideoGraph: Recognizing Minutes-Long Human Activities in Videoscited by 57
  20. 2019Weakly-supervised Action Localization with Background Modelingcited by 53
  21. 2019Weakly Supervised Energy-Based Learning for Action Segmentationcited by 53
  22. 20193C-Net: Category Count and Center Loss for Weakly-Supervised Action Localizationcited by 52
  23. 2015Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Datacited by 40
  24. 2020Set-Constrained Viterbi for Set-Supervised Action Segmentationcited by 35
  25. 2015What's Cookin'? Interpreting Cooking Videos using Text, Speech and Visioncited by 31
  26. 2019Fast Weakly Supervised Action Segmentation Using Mutual Consistencycited by 30
  27. 2019Action Modifiers: Learning from Adverbs in Instructional Videoscited by 27
  28. 2021Learning Discriminative Prototypes with Dynamic Time Warpingcited by 23
  29. 2020Action Graphs: Weakly-supervised Action Localization with Graph Convolution Networkscited by 16
  30. 2021FIFA: Fast Inference Approximation for Action Segmentationcited by 16
  31. 2020On Evaluating Weakly Supervised Action Segmentation Methodscited by 4
  32. 2020Action Duration Prediction for Segment-Level Alignment of Weakly-Labeled Videoscited by 3

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 2; 2024: 1; 2025: 2.

11 papers.

  1. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  2. 2022Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Videocited by 43
  3. 2021SVIP: Sequence VerIfication for Procedures in Videoscited by 16
  4. 2023Skip-Plan: Procedure Planning in Instructional Videos via Condensed Action Space Learningcited by 14
  5. 2022Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videoscited by 12
  6. 2022Multimedia Generative Script Learning for Task Planningcited by 12
  7. 2022Multi-Task Learning of Object State Changes from Uncurated Videoscited by 10
  8. 2022Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actionscited by 8
  9. 2024Learning to Visually Connect Actions and their Effectscited by 2
  10. 2025Pose-Aware Weakly-Supervised Action Segmentationcited by 2
  11. 2025Ask2Loc: Learning to Locate Instructional Visual Answers by Asking Questionscited by 2