# Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows

arXiv: https://arxiv.org/abs/2209.05840 · HTML: https://mlsanity.com/paper/2209.05840/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 13 Sep 2022.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Learning a Text-Video Embedding from Incomplete and Heterogeneous Data](https://mlsanity.com/paper/1804.02516/) (2018)
- [RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes](https://mlsanity.com/paper/1809.00812/) (2018)
- [Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension](https://mlsanity.com/paper/1805.06975/) (2018)
- [Simulating Action Dynamics with Neural Process Networks](https://mlsanity.com/paper/1711.05313/) (2017)
- [A Dataset for Tracking Entities in Open Domain Procedural Text](https://mlsanity.com/paper/2011.08092/) (2020)
- [Learning Program Representations for Food Images and Cooking Recipes](https://mlsanity.com/paper/2203.16071/) (2022)
- [Multi-modal Cooking Workflow Construction for Food Recipes](https://mlsanity.com/paper/2008.09151/) (2020)
- [CITE: A Corpus of Image-Text Discourse Relations](https://mlsanity.com/paper/1904.06286/) (2019)

## Built on by (most cited first, 8 of 8)

- [SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.01599/) (2024)
- [J-CRe3: A Japanese Conversation Dataset for Real-world Reference Resolution](https://mlsanity.com/paper/2403.19259/) (2024)
- [COM Kitchens: An Unedited Overhead-view Video Dataset as a Vision-Language Benchmark](https://mlsanity.com/paper/2408.02272/) (2024)
- [STATUS Bench: A Rigorous Benchmark for Evaluating Object State Understanding in Vision-Language Models](https://mlsanity.com/paper/2510.22571/) (2025)
- [MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models](https://mlsanity.com/paper/2502.10886/) (2025)
- [Towards Flow Graph Prediction of Open-Domain Procedural Texts](https://mlsanity.com/paper/2305.19497/) (2023)
- [Towards Multimodal Graph Large Language Model](https://mlsanity.com/paper/2506.09738/) (2025)
- [ReGraph: Learning to Generate Recipe Graphs from Food Images](https://mlsanity.com/paper/2608.06917/) (2026)

## Abstract (arXiv)

We present a new multimodal dataset called Visual Recipe Flow, which enables us to learn each cooking action result in a recipe text. The dataset consists of object state changes and the workflow of the recipe text. The state change is represented as an image pair, while the workflow is represented as a recipe flow graph (r-FG). The image pairs are grounded in the r-FG, which provides the cross-modal relation. With our dataset, one can try a range of applications, from multimodal commonsense reasoning and procedural text generation.