# A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks

arXiv: https://arxiv.org/abs/2005.09606 · HTML: https://mlsanity.com/paper/2005.09606/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2020.

## Built on by (most cited first, 15 of 15)

- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans](https://mlsanity.com/paper/2406.15823/) (2024)
- [Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions](https://mlsanity.com/paper/2106.03158/) (2021)
- [Reasoning about Procedures with Natural Language Processing: A Tutorial](https://mlsanity.com/paper/2205.07455/) (2022)
- [Cultural Adaptation of Recipes](https://mlsanity.com/paper/2310.17353/) (2023)
- [Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish](https://mlsanity.com/paper/2309.06698/) (2023)
- [OSCAR: Object Status and Contextual Awareness for Recipes to Support Non-Visual Cooking](https://mlsanity.com/paper/2503.05962/) (2025)
- [PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset](https://mlsanity.com/paper/2403.03167/) (2024)
- [Contrastive Sequential-Diffusion Learning: Non-linear and Multi-Scene Instructional Video Synthesis](https://mlsanity.com/paper/2407.11814/) (2024)
- [Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension](https://mlsanity.com/paper/2204.02566/) (2022)
- [Human in the loop approaches in multi-modal conversational task guidance system development](https://mlsanity.com/paper/2211.01824/) (2022)
- [Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking](https://mlsanity.com/paper/2507.03330/) (2025)
- [Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA](https://mlsanity.com/paper/2605.14928/) (2026)
- [CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text](https://mlsanity.com/paper/2310.14326/) (2023)
- [Towards Multimodal Graph Large Language Model](https://mlsanity.com/paper/2506.09738/) (2025)

## Abstract (arXiv)

Many high-level procedural tasks can be decomposed into sequences of instructions that vary in their order and choice of tools. In the cooking domain, the web offers many partially-overlapping text and video recipes (i.e. procedures) that describe how to make the same dish (i.e. high-level task). Aligning instructions for the same dish across different sources can yield descriptive visual explanations that are far richer semantically than conventional textual instructions, providing commonsense insight into how real-world procedures are structured. Learning to align these different instruction sets is challenging because: a) different recipes vary in their order of instructions and use of ingredients; and b) video instructions can be noisy and tend to contain far more information than text instructions. To address these challenges, we first use an unsupervised alignment algorithm that learns pairwise alignments between instructions of different recipes for the same dish. We then use a graph algorithm to derive a joint alignment between multiple text and multiple video recipes for the same dish. We release the Microsoft Research Multimodal Aligned Recipe Corpus containing 150K pairwise alignments between recipes across 4,262 dishes with rich commonsense information.