mlsanityData to 14 Oct 2026

Visual Goal-Step Inference using wikiHow

arXiv 2104.05845 ↗First posted 12 Apr 2021cs.CV cs.AI cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

Understanding what sequence of steps are needed to complete a goal can help artificial intelligence systems reason about human activities. Past work in NLP has examined the task of goal-step inference for text. We introduce the visual analogue. We propose the Visual Goal-Step Inference (VGSI) task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal. With a new dataset harvested from wikiHow consisting of 772,277 images representing human actions, we show that our task is challenging for state-of-the-art multimodal models. Moreover, the multimodal representation learned from our data can be effectively transferred to other datasets like HowTo100m, increasing the VGSI accuracy by 15 - 20%. Our task will facilitate multimodal reasoning about procedural events.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

22 of 30 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2017Billion-scale similarity search with GPUscited by 2,497
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  11. 2014Deep metric learning using Triplet networkcited by 422
  12. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
10 more
  1. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  2. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  3. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  4. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  5. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  6. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  7. 2018Joint Slot Filling and Intent Detection via Capsule Neural Networkscited by 73
  8. 2020Reasoning about Goals, Steps, and Temporal Ordering with WikiHowcited by 66
  9. 2020Intent Detection with WikiHowcited by 19
  10. 2019Learning Household Task Knowledge from WikiHow Descriptionscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 9; 2024: 12; 2025: 8; 2026: 2.

43 papers.

Sort:

By area:

  1. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  2. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 51
  3. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  4. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  5. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 33
  6. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 31
  7. 2024Holistic Evaluation for Interleaved Text-and-Image Generationcited by 28
  8. 2023Causal Reasoning of Entities and Events in Procedural Textscited by 23
  9. 2022Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Datacited by 22
  10. 2021Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manualscited by 18
  11. 2021Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrievalcited by 13
  12. 2022Multimedia Generative Script Learning for Task Planningcited by 12
31 more
  1. 2022Reasoning about Actions over Visual and Linguistic Modalities: A Surveycited by 10
  2. 2022Coalescing Global and Local Information for Procedural Text Understandingcited by 10
  3. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9
  4. 2022Reasoning about Procedures with Natural Language Processing: A Tutorialcited by 8
  5. 2023OpenPI2.0: An Improved Dataset for Entity Tracking in Textscited by 8
  6. 2024ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructionscited by 8
  7. 2023Generating Illustrated Instructionscited by 7
  8. 2022Evaluating and Improving Factuality in Multimodal Abstractive Summarizationcited by 6
  9. 2024Many-to-many Image Generation with Auto-regressive Diffusion Modelscited by 6
  10. 2023Rewriting the Script: Adapting Text Instructions for Voice Interactioncited by 5
  11. 2023Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkishcited by 5
  12. 2022Schema-Guided Event Graph Completioncited by 4
  13. 2023Non-Sequential Graph Script Induction via Multimedia Groundingcited by 4
  14. 2024WorldAPIs: The World Is Worth How Many APIs? A Thought Experimentcited by 4
  15. 2025Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectioncited by 4
  16. 2022Incorporating Task-specific Concept Knowledge into Script Learningcited by 3
  17. 2024Coherent Zero-Shot Visual Instruction Generationcited by 3
  18. 2025A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generationcited by 3
  19. 2023MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Taskscited by 2
  20. 2024ActionCOMET: A Zero-shot Approach to Learn Image-specific Commonsense Concepts about Actionscited by 2
  21. 2025Towards Text-Image Interleaved Retrievalcited by 2
  22. 2025Enhance Multimodal Consistency and Coherence for Text-Image Plan Generationcited by 2
  23. 2026Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QAcited by 2
  24. 2022Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Predictioncited by 1
  25. 2024VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Taskscited by 1
  26. 2025RecipeGen: A Benchmark for Real-World Recipe Image Generationcited by 1
  27. 2025I2G: Generating Instructional Illustrations via Text-Conditioned Diffusioncited by 1
  28. 2025CookAnything: A Framework for Flexible and Consistent Multi-Step Recipe Image Generationcited by 1
  29. 2022Learning Action-Effect Dynamics from Pairs of Scene-graphscited by 0
  30. 2025RecipeGen: A Step-Aligned Multimodal Benchmark for Real-World Recipe Generationcited by 0
  31. 2026InstructionCrafter: Generating Consistent and High-Fidelity Visual Instructionscited by 0