mlsanityData to 7 Oct 2026

What's Cookin'? Interpreting Cooking Videos using Text, Speech and Vision

arXiv 1503.01558 ↗First posted 5 Mar 2015cs.CL cs.CV cs.IRcited by 31

Not on the map. Why?

Abstract (arXiv)

We present a novel method for aligning a sequence of instructions to a video of someone carrying out a task. In particular, we focus on the cooking domain, where the instructions correspond to the recipe. Our technique relies on an HMM to align the recipe steps to the (automatically generated) speech transcript. We then refine this alignment using a state-of-the-art visual food detector, based on a deep convolutional neural network. We show that our technique outperforms simpler techniques based on keyword spotting. It also enables interesting applications, such as automatically illustrating recipes with keyframes, and searching within a video for events of interest.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Mar 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 7; 2023: 8; 2024: 1; 2025: 2; 2026: 2.

31 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  3. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  4. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  5. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  6. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 103
  7. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  8. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  9. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 48
  10. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  11. 2023Learning Procedure-aware Video Representation from Instructional Videos and Their Narrationscited by 46
  12. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
19 more
  1. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  2. 2021Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activitiescited by 26
  3. 2022Timestamp-Supervised Action Segmentation with Graph Convolutional Networkscited by 18
  4. 2023Every Mistake Counts in Assemblycited by 18
  5. 2021Learning Temporal Dynamics from Cycles in Narrated Videocited by 13
  6. 2023STEPs: Self-Supervised Key Step Extraction and Localization from Unlabeled Procedural Videoscited by 13
  7. 2023United We Stand, Divided We Fall: UnityGraph for Unsupervised Procedure Learning from Videoscited by 13
  8. 2024A Survey on Integrated Sensing, Communication, and Computationcited by 12
  9. 2021Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videoscited by 11
  10. 2023OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentationcited by 11
  11. 2021Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actionscited by 10
  12. 2021Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videoscited by 9
  13. 2022Rethinking Cooking State Recognition with Vision Transformerscited by 9
  14. 2021CoSeg: Cognitively Inspired Unsupervised Generic Event Segmentationcited by 7
  15. 2025Procedure Learning via Regularized Gromov-Wasserstein Optimal Transportcited by 4
  16. 2025REMAP: Regularized Matching and Partial Alignment of Video Embeddingscited by 1
  17. 2026Hierarchical Action Learning for Weakly-Supervised Action Segmentationcited by 1
  18. 2021Reconstructing and grounding narrated instructional videos in 3Dcited by 0
  19. 2026How Does India Cook Biryani?cited by 0