# Visual Goal-Step Inference using wikiHow

arXiv: https://arxiv.org/abs/2104.05845 · HTML: https://mlsanity.com/paper/2104.05845/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [Learning Two-Branch Neural Networks for Image-Text Matching Tasks](https://mlsanity.com/paper/1704.03470/) (2017)
- [Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge](https://mlsanity.com/paper/1609.06647/) (2016)
- [Deep Fragment Embeddings for Bidirectional Image Sentence Mapping](https://mlsanity.com/paper/1406.5679/) (2014)
- [Joint Slot Filling and Intent Detection via Capsule Neural Networks](https://mlsanity.com/paper/1812.09471/) (2018)
- [Reasoning about Goals, Steps, and Temporal Ordering with WikiHow](https://mlsanity.com/paper/2009.07690/) (2020)
- [Intent Detection with WikiHow](https://mlsanity.com/paper/2009.05781/) (2020)
- [Learning Household Task Knowledge from WikiHow Descriptions](https://mlsanity.com/paper/1909.06414/) (2019)

## Built on by (most cited first, 30 of 43)

- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities](https://mlsanity.com/paper/2312.14556/) (2023)
- [Procedure-Aware Pretraining for Instructional Video Understanding](https://mlsanity.com/paper/2303.18230/) (2023)
- [MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models](https://mlsanity.com/paper/2406.07594/) (2024)
- [MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2410.10139/) (2024)
- [OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation](https://mlsanity.com/paper/2411.18499/) (2024)
- [Holistic Evaluation for Interleaved Text-and-Image Generation](https://mlsanity.com/paper/2406.14643/) (2024)
- [Causal Reasoning of Entities and Events in Procedural Texts](https://mlsanity.com/paper/2301.10896/) (2023)
- [Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data](https://mlsanity.com/paper/2203.07264/) (2022)
- [Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals](https://mlsanity.com/paper/2110.08486/) (2021)
- [Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrieval](https://mlsanity.com/paper/2111.09276/) (2021)
- [Multimedia Generative Script Learning for Task Planning](https://mlsanity.com/paper/2208.12306/) (2022)
- [Reasoning about Actions over Visual and Linguistic Modalities: A Survey](https://mlsanity.com/paper/2207.07568/) (2022)
- [Coalescing Global and Local Information for Procedural Text Understanding](https://mlsanity.com/paper/2208.12848/) (2022)
- [Modality-Specialized Synergizers for Interleaved Vision-Language Generalists](https://mlsanity.com/paper/2407.03604/) (2024)
- [Reasoning about Procedures with Natural Language Processing: A Tutorial](https://mlsanity.com/paper/2205.07455/) (2022)
- [OpenPI2.0: An Improved Dataset for Entity Tracking in Texts](https://mlsanity.com/paper/2305.14603/) (2023)
- [ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions](https://mlsanity.com/paper/2412.01987/) (2024)
- [Generating Illustrated Instructions](https://mlsanity.com/paper/2312.04552/) (2023)
- [Evaluating and Improving Factuality in Multimodal Abstractive Summarization](https://mlsanity.com/paper/2211.02580/) (2022)
- [Many-to-many Image Generation with Auto-regressive Diffusion Models](https://mlsanity.com/paper/2404.03109/) (2024)
- [Rewriting the Script: Adapting Text Instructions for Voice Interaction](https://mlsanity.com/paper/2306.09992/) (2023)
- [Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish](https://mlsanity.com/paper/2309.06698/) (2023)
- [Schema-Guided Event Graph Completion](https://mlsanity.com/paper/2206.02921/) (2022)
- [Non-Sequential Graph Script Induction via Multimedia Grounding](https://mlsanity.com/paper/2305.17542/) (2023)
- [WorldAPIs: The World Is Worth How Many APIs? A Thought Experiment](https://mlsanity.com/paper/2407.07778/) (2024)
- [Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflection](https://mlsanity.com/paper/2503.13500/) (2025)
- [Incorporating Task-specific Concept Knowledge into Script Learning](https://mlsanity.com/paper/2209.00068/) (2022)
- [Coherent Zero-Shot Visual Instruction Generation](https://mlsanity.com/paper/2406.04337/) (2024)
- [A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation](https://mlsanity.com/paper/2506.09427/) (2025)

## Abstract (arXiv)

Understanding what sequence of steps are needed to complete a goal can help artificial intelligence systems reason about human activities. Past work in NLP has examined the task of goal-step inference for text. We introduce the visual analogue. We propose the Visual Goal-Step Inference (VGSI) task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal. With a new dataset harvested from wikiHow consisting of 772,277 images representing human actions, we show that our task is challenging for state-of-the-art multimodal models. Moreover, the multimodal representation learned from our data can be effectively transferred to other datasets like HowTo100m, increasing the VGSI accuracy by 15 - 20%. Our task will facilitate multimodal reasoning about procedural events.