mlsanityData to 7 Oct 2026

Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videos

arXiv 2203.13309 ↗First posted 24 Mar 2022cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper addresses a new problem of weakly-supervised online action segmentation in instructional videos. We present a framework to segment streaming videos online at test time using Dynamic Programming and show its advantages over greedy sliding window approach. We improve our framework by introducing the Online-Offline Discrepancy Loss (OODL) to encourage the segmentation results to have a higher temporal consistency. Furthermore, only during training, we exploit frame-wise correspondence between multiple views as supervision for training weakly-labeled instructional videos. In particular, we investigate three different multi-view inference techniques to generate more accurate frame-wise pseudo ground-truth with no additional annotation cost. We present results and ablation studies on two benchmark multi-view datasets, Breakfast and IKEA ASM. Experimental results show efficacy of the proposed methods both qualitatively and quantitatively in two domains of cooking and assembly.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 24 Mar 2022.

Builds on

43 of 60 references.

References · 43

  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  2. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  3. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  4. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  5. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  6. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  7. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 102
  8. 2019What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessmentcited by 100
  9. 2020The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Posecited by 95
  10. 2019Procedure Planning in Instructional Videoscited by 92
  11. 2021Long Short-Term Transformer for Online Action Detectioncited by 87
  12. 2020Alleviating Over-segmentation Errors by Detecting Action Boundariescited by 86
31 more
  1. 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
  2. 2018Temporal Recurrent Networks for Online Action Detectioncited by 79
  3. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 77
  4. 2018Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignmentcited by 74
  5. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
  6. 2018NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learningcited by 67
  7. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  8. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  9. 2019Weakly Supervised Energy-Based Learning for Action Segmentationcited by 53
  10. 2021Temporal Action Segmentation from Timestamp Supervisioncited by 48
  11. 2021Learning by Aligning Videos in Timecited by 47
  12. 2021Global2Local: Efficient Structure Search for Video Action Segmentationcited by 42
  13. 2015Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Datacited by 40
  14. 2019Uncertainty-Aware Anticipation of Activitiescited by 38
  15. 2019Learning to Discriminate Information for Online Action Detectioncited by 37
  16. 2021Multiview Pseudo-Labeling for Semi-supervised Learning from Videocited by 32
  17. 2019Fast Weakly Supervised Action Segmentation Using Mutual Consistencycited by 30
  18. 2021Learning Discriminative Prototypes with Dynamic Time Warpingcited by 23
  19. 2019StartNet: Online Detection of Action Start in Untrimmed Videoscited by 21
  20. 2020WOAD: Weakly Supervised Online Action Detection in Untrimmed Videoscited by 19
  21. 2017Identifying First-person Camera Wearers in Third-person Videoscited by 18
  22. 2020Privileged Knowledge Distillation for Online Action Detectioncited by 18
  23. 2021Recognizing Actions in Videos from Unseen Viewpointscited by 18
  24. 2018Online Detection of Action Start in Untrimmed, Streaming Videoscited by 17
  25. 2021FIFA: Fast Inference Approximation for Action Segmentationcited by 16
  26. 2017Summarizing First-Person Videos from Third Persons' Points of Viewscited by 15
  27. 2020LAP-Net: Adaptive Features Sampling via Learning Action Progression for Online Action Detectioncited by 13
  28. 2021Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videoscited by 11
  29. 2017Multi-View Surveillance Video Summarization via Joint Embedding and Sparse Optimizationcited by 5
  30. 2020Action Duration Prediction for Segment-Level Alignment of Weakly-Labeled Videoscited by 3
  31. 2020Online Action Detection in Streaming Videos with Time Bufferscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 3; 2025: 3; 2026: 1.

12 papers.

  1. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  2. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  3. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  4. 2023Skip-Plan: Procedure Planning in Instructional Videos via Condensed Action Space Learningcited by 14
  5. 2024Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignmentcited by 13
  6. 2024OnlineTAS: An Online Baseline for Temporal Action Segmentationcited by 7
  7. 2025Pose-Aware Weakly-Supervised Action Segmentationcited by 2
  8. 2023End-to-End Streaming Video Temporal Action Segmentation with Reinforce Learningcited by 1
  9. 2024O-TALC: Steps Towards Combating Oversegmentation within Online Action Segmentationcited by 1
  10. 2026MistExit: Learning to Exit for Early Mistake Detection in Procedural Videoscited by 1
  11. 2025Towards Generalizing Temporal Action Segmentation to Unseen Viewscited by 0
  12. 2025Probabilistic Temporal Masked Attention for Cross-view Online Action Detectioncited by 0