mlsanityData to 7 Oct 2026

Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions

arXiv 2106.03158 ↗First posted 6 Jun 2021cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Can we teach a robot to recognize and make predictions for activities that it has never seen before? We tackle this problem by learning models for video from text. This paper presents a hierarchical model that generalizes instructional knowledge from large-scale text corpora and transfers the knowledge to video. Given a portion of an instructional video, our model recognizes and predicts coherent and plausible actions multiple steps into the future, all in rich natural language. To demonstrate the capabilities of our model, we introduce the Tasty Videos Dataset V2, a collection of 4022 recipes for zero-shot learning, recognition and anticipation. Extensive experiments with various evaluation metrics demonstrate the potential of our method for generalization, given limited video data for training models.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 64 of its 105 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

64 of 105 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2016Layer Normalizationcited by 5,579
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
38 more
  1. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  2. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  3. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  4. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  5. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 462
  6. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  7. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  8. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  9. 2015Skip-Thought Vectorscited by 321
  10. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  11. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  12. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  13. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  14. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  15. 2015A Neural Conversational Modelcited by 195
  16. 2018Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Imagescited by 118
  17. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  18. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 106
  19. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  20. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  21. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  22. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  23. 2015Describing Videos by Exploiting Temporal Structurecited by 88
  24. 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
  25. 2020Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applicationscited by 79
  26. 2016Segmental Spatiotemporal CNNs for Fine-grained Action Segmentationcited by 78
  27. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 77
  28. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  29. 2016Connectionist Temporal Modeling for Weakly Supervised Action Labelingcited by 70
  30. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
  31. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  32. 2018Jointly Localizing and Describing Events for Dense Video Captioningcited by 60
  33. 2018Inverse Cooking: Recipe Generation from Food Imagescited by 47
  34. 2016Learning Attributes Equals Multi-Source Domain Generalizationcited by 44
  35. 2018Towards Universal Representation for Unseen Action Recognitioncited by 44
  36. 2018Zero-Shot Anticipation for Instructional Activitiescited by 43
  37. 2019Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Imagescited by 38
  38. 2018Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddingscited by 35

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 3; 2026: 1.

10 papers.

  1. 2023Pretrained Language Models as Visual Planners for Human Assistancecited by 28
  2. 2023A Review of Deep Learning for Video Captioningcited by 28
  3. 2023Every Mistake Counts in Assemblycited by 18
  4. 2024Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videoscited by 15
  5. 2025Context-Enhanced Memory-Refined Transformer for Online Action Detectioncited by 13
  6. 2023GePSAn: Generative Procedure Step Anticipation in Cooking Videoscited by 9
  7. 2024Human Action Anticipation: A Surveycited by 8
  8. 2025Stitch-a-Demo: Video Demonstrations from Multistep Descriptionscited by 1
  9. 2025Predicting Implicit Arguments in Procedural Video Instructionscited by 1
  10. 2026Q-Tag: Watermarking Quantum Circuit Generative Modelscited by 0