mlsanityData to 14 Oct 2026

Towards Automatic Learning of Procedures from Web Instructional Videos

arXiv 1703.09788 ↗First posted 28 Mar 2017cs.CVcited by 619

Not on the map. Why?

Cited by 114 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The potential for agents, whether embodied or software, to learn by observing other agents performing procedures involving objects and actions is rich. Current research on automatic procedure learning heavily relies on action labels or video subtitles, even during the evaluation phase, which makes them infeasible in real-world scenarios. This leads to our question: can the human-consensus structure of a procedure be learned from a large set of long, unconstrained videos (e.g., instructional videos from YouTube) with only visual evidence? To answer this question, we introduce the problem of procedure segmentation--to segment a video procedure into category-independent procedure segments. Given that no large-scale dataset is available for this problem, we collect a large-scale procedure segmentation dataset with procedure segments temporally localized and described; we use cooking videos and name the dataset YouCook2. We propose a segment-level recurrent network for generating procedure segments by modeling the dependencies across segments. The generated segments can be used as pre-processing for other tasks, such as dense video captioning and event parsing. We show in our experiments that the proposed model outperforms competitive baselines in procedure segmentation.

Key facts. Cited by 619 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 69; 2022: 83; 2023: 111; 2024: 124; 2025: 149; 2026: 83.

619 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  5. 2021Transformers in Vision: A Surveycited by 1,013
  6. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  8. 2023Segment Everything Everywhere All at Oncecited by 548
  9. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  10. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  11. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  12. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
38 more
  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  2. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  3. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  4. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  5. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  6. 2022Multimodal Learning with Transformers: A Surveycited by 400
  7. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  8. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  9. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  10. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  11. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  12. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
  13. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  14. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  15. 2022Egocentric Video-Language Pretrainingcited by 232
  16. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  17. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  18. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  19. 2023A Survey on Video Diffusion Modelscited by 206
  20. 2023Video Understanding with Large Language Models: A Surveycited by 206
  21. 2022Learning Video Representations from Large Language Modelscited by 205
  22. 2023Multimodal Large Language Models: A Surveycited by 195
  23. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  24. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  25. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  26. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 162
  27. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  28. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  29. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  30. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  31. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  32. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  33. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  34. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 135
  35. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  36. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  37. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  38. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 128

50 most cited of 619 All 619 (JSON)