mlsanityData to 7 Oct 2026

Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrieval

arXiv 2111.09276 ↗First posted 17 Nov 2021cs.CV cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Schemata are structured representations of complex tasks that can aid artificial intelligence by allowing models to break down complex tasks into intermediate steps. We propose a novel system that induces schemata from web videos and generalizes them to capture unseen tasks with the goal of improving video retrieval performance. Our system proceeds in three major phases: (1) Given a task with related videos, we construct an initial schema for a task using a joint video-text model to match video segments with text representing steps from wikiHow; (2) We generalize schemata to unseen tasks by leveraging language models to edit the text within existing schemata. Through generalization, we can allow our schemata to cover a more extensive range of tasks with a small amount of learning data; (3) We conduct zero-shot instructional video retrieval with the unseen task names as the queries. Our schema-guided approach outperforms existing methods for video retrieval, and we demonstrate that the schemata induced by our system are better than those generated by other models.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 17 Nov 2021.

Builds on

36 of 59 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  7. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  8. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  9. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  10. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  11. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  12. 2020Multi-modal Transformer for Video Retrievalcited by 395
24 more
  1. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  2. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  3. 2018Graph R-CNN for Scene Graph Generationcited by 296
  4. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  5. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  6. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  7. 2019Learning by Abstraction: The Neural State Machinecited by 113
  8. 2018Neural Baby Talkcited by 107
  9. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  10. 2019Scene Graph Generation with External Knowledge and Image Reconstructioncited by 102
  11. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  12. 2020Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphscited by 87
  13. 2018Dual Encoding for Zero-Example Video Retrievalcited by 85
  14. 2019Unpaired Image Captioning via Scene Graph Alignmentscited by 82
  15. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  16. 2020Reasoning about Goals, Steps, and Temporal Ordering with WikiHowcited by 66
  17. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  18. 2018Zero-Shot Anticipation for Instructional Activitiescited by 43
  19. 2021Visual Goal-Step Inference using wikiHowcited by 42
  20. 2021Goal-Oriented Script Constructioncited by 30
  21. 2020Analogous Process Structure Induction for Sub-event Sequence Predictioncited by 26
  22. 2021ReFormer: The Relational Transformer for Image Captioningcited by 23
  23. 2020A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videoscited by 11
  24. 2018Hierarchical Quantized Representations for Script Generationcited by 10

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 8; 2024: 2.

13 papers.

Sort:

By area:

  1. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  2. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  3. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  4. 2023Causal Reasoning of Entities and Events in Procedural Textscited by 23
  5. 2022Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Datacited by 22
  6. 2024Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videoscited by 15
  7. 2022Multimedia Generative Script Learning for Task Planningcited by 12
  8. 2023OpenPI2.0: An Improved Dataset for Entity Tracking in Textscited by 8
  9. 2023VideoPro: A Visual Analytics Approach for Interactive Video Programmingcited by 7
  10. 2023Rewriting the Script: Adapting Text Instructions for Voice Interactioncited by 5
  11. 2023Non-Sequential Graph Script Induction via Multimedia Groundingcited by 4
  12. 2022Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Predictioncited by 1
1 more
  1. 2024Chain-of-Specificity: An Iteratively Refining Method for Eliciting Knowledge from Large Language Modelscited by 0