mlsanityData to 7 Oct 2026

Cross-task weakly supervised learning from instructional videos

arXiv 1903.08225 ↗First posted 19 Mar 2019cs.CVcited by 199

Not on the map. Why?

Abstract (arXiv)

In this paper we investigate learning visual models for the steps of ordinary tasks using weak supervision via instructional narrations and an ordered list of steps instead of strong supervision via temporal annotations. At the heart of our approach is the observation that weakly supervised learning may be easier if a model shares components while learning different steps: `pour egg' should be trained jointly with other tasks involving `pour' and `egg'. We formalize this in a component model for recognizing steps and a weakly supervised learning framework that can learn this model under temporal constraints from narration and the list of steps. Past data does not permit systematic studying of sharing and so we also gather a new dataset, CrossTask, aimed at assessing cross-task sharing. Our experiments demonstrate that sharing across tasks improves performance, especially when done at the component level and that our component model can parse previously unseen tasks by virtue of its compositionality.

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 34; 2023: 43; 2024: 40; 2025: 35; 2026: 27.

199 papers.

Sort:

By area:

  1. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  2. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  3. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  4. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  5. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  6. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 103
  7. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  8. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  9. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  10. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  11. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  12. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
38 more
  1. 2022Temporal Alignment Networks for Long-term Videocited by 84
  2. 2023Hierarchical Video-Moment Retrieval and Step-Captioningcited by 83
  3. 2021PixelSynth: Generating a 3D-Consistent Experience from a Single Imagecited by 82
  4. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  5. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  6. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  7. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 79
  8. 2021Object-Region Video Transformerscited by 73
  9. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  10. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  11. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  12. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
  13. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
  14. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
  15. 2022Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learningcited by 59
  16. 2021Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learningcited by 53
  17. 2022P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervisioncited by 52
  18. 2022Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videoscited by 50
  19. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  20. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  21. 2023Learning Procedure-aware Video Representation from Instructional Videos and Their Narrationscited by 46
  22. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  23. 2023IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Settingcited by 46
  24. 2022LocVTP: Video-Text Pre-training for Temporal Localizationcited by 44
  25. 2022Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Videocited by 43
  26. 2021Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outlierscited by 39
  27. 2022Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videoscited by 39
  28. 2023StepFormer: Self-supervised Step Discovery and Localization in Instructional Videoscited by 37
  29. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  30. 2023PDPP: Projected Diffusion for Procedure Planning in Instructional Videoscited by 36
  31. 2024PREGO: online mistake detection in PRocedural EGOcentric videoscited by 36
  32. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  33. 2022TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliencycited by 35
  34. 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
  35. 2023Pretrained Language Models as Visual Planners for Human Assistancecited by 28
  36. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  37. 2021Survey: Transformer based Video-Language Pre-trainingcited by 27
  38. 2023Learning and Verification of Task Structure in Instructional Videoscited by 26

50 most cited of 199 All 199 (JSON)