# Multi-modal Cooking Workflow Construction for Food Recipes

arXiv: https://arxiv.org/abs/2008.09151 · HTML: https://mlsanity.com/paper/2008.09151/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2020.

## Built on by (most cited first, 15 of 15)

- [CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans](https://mlsanity.com/paper/2406.15823/) (2024)
- [RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models](https://mlsanity.com/paper/2407.12730/) (2024)
- [Constructing Flow Graphs from Procedural Cybersecurity Texts](https://mlsanity.com/paper/2105.14357/) (2021)
- [Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows](https://mlsanity.com/paper/2209.05840/) (2022)
- [Action Dynamics Task Graphs for Learning Plannable Representations of Procedural Tasks](https://mlsanity.com/paper/2302.05330/) (2023)
- [Counterfactual Recipe Generation: Exploring Compositional Generalization in a Realistic Scenario](https://mlsanity.com/paper/2210.11431/) (2022)
- [Haste Makes Waste: Evaluating Planning Abilities of LLMs for Efficient and Feasible Multitasking with Time Constraints Between Actions](https://mlsanity.com/paper/2503.02238/) (2025)
- [A Rich Recipe Representation as Plan to Support Expressive Multi Modal Queries on Recipe Content and Preparation Process](https://mlsanity.com/paper/2203.17109/) (2022)
- [Recipe Generation from Unsegmented Cooking Videos](https://mlsanity.com/paper/2209.10134/) (2022)
- [NoteIt: A System Converting Instructional Videos to Interactable Notes Through Multimodal Video Understanding](https://mlsanity.com/paper/2508.14395/) (2025)
- [CookAnything: A Framework for Flexible and Consistent Multi-Step Recipe Image Generation](https://mlsanity.com/paper/2512.03540/) (2025)
- [Towards Flow Graph Prediction of Open-Domain Procedural Texts](https://mlsanity.com/paper/2305.19497/) (2023)
- [A Graphical Formalism for Commonsense Reasoning with Recipes](https://mlsanity.com/paper/2306.09042/) (2023)
- [Box2Flow: Instance-based Action Flow Graphs from Videos](https://mlsanity.com/paper/2409.00295/) (2024)
- [A Highly Clean Recipe Dataset with Ingredient States Annotation for State Probing Task](https://mlsanity.com/paper/2507.17232/) (2025)

## Abstract (arXiv)

Understanding food recipe requires anticipating the implicit causal effects of cooking actions, such that the recipe can be converted into a graph describing the temporal workflow of the recipe. This is a non-trivial task that involves common-sense reasoning. However, existing efforts rely on hand-crafted features to extract the workflow graph from recipes due to the lack of large-scale labeled datasets. Moreover, they fail to utilize the cooking images, which constitute an important part of food recipes. In this paper, we build MM-ReS, the first large-scale dataset for cooking workflow construction, consisting of 9,850 recipes with human-labeled workflow graphs. Cooking steps are multi-modal, featuring both text instructions and cooking images. We then propose a neural encoder-decoder model that utilizes both visual and textual information to construct the cooking workflow, which achieved over 20% performance gain over existing hand-crafted baselines.