# CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion

arXiv: https://arxiv.org/abs/2501.09042 · HTML: https://mlsanity.com/paper/2501.09042/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 41 of its 66 references that are in the map.
- First posted on arXiv 15 Jan 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [Video Probabilistic Diffusion Models in Projected Latent Space](https://mlsanity.com/paper/2302.07685/) (2023)
- [NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation](https://mlsanity.com/paper/2303.12346/) (2023)
- [Generating Images from Captions with Attention](https://mlsanity.com/paper/1511.02793/) (2015)
- [Learning What and Where to Draw](https://mlsanity.com/paper/1610.02454/) (2016)
- [Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspective](https://mlsanity.com/paper/2408.06741/) (2024)
- [Make-A-Story: Visual Memory Conditioned Consistent Story Generation](https://mlsanity.com/paper/2211.13319/) (2022)
- [A Survey on Food Computing](https://mlsanity.com/paper/1808.07202/) (2018)
- [Inverse Cooking: Recipe Generation from Food Images](https://mlsanity.com/paper/1812.06164/) (2018)
- [Multimodal Procedural Planning via Dual Text-Image Prompting](https://mlsanity.com/paper/2305.01795/) (2023)
- [FoodLMM: A Versatile Food Assistant using Large Multi-modal Model](https://mlsanity.com/paper/2312.14991/) (2023)

## Built on by (most cited first, 4 of 4)

- [Chain-of-Cooking:Cooking Process Visualization via Bidirectional Chain-of-Thought Guidance](https://mlsanity.com/paper/2507.21529/) (2025)
- [Thinking with Frames: Generative Video Distortion Evaluation via Frame Reward Model](https://mlsanity.com/paper/2601.04033/) (2026)
- [Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model](https://mlsanity.com/paper/2608.05976/) (2026)
- [Real-Time Cooked Food Image Synthesis and Visual Cooking Progress Monitoring on Edge Devices](https://mlsanity.com/paper/2511.16965/) (2025)

## Abstract (arXiv)

Recent advancements in text-to-image generation models have excelled in creating diverse and realistic images. This success extends to food imagery, where various conditional inputs like cooking styles, ingredients, and recipes are utilized. However, a yet-unexplored challenge is generating a sequence of procedural images based on cooking steps from a recipe. This could enhance the cooking experience with visual guidance and possibly lead to an intelligent cooking simulation system. To fill this gap, we introduce a novel task called cooking procedural image generation. This task is inherently demanding, as it strives to create photo-realistic images that align with cooking steps while preserving sequential consistency. To collectively tackle these challenges, we present CookingDiffusion, a novel approach that leverages Stable Diffusion and three innovative Memory Nets to model procedural prompts. These prompts encompass text prompts (representing cooking steps), image prompts (corresponding to cooking images), and multi-modal prompts (mixing cooking steps and images), ensuring the consistent generation of cooking procedural images. To validate the effectiveness of our approach, we preprocess the YouCookII dataset, establishing a new benchmark. Our experimental results demonstrate that our model excels at generating high-quality cooking procedural images with remarkable consistency across sequential cooking steps, as measured by both the FID and the proposed Average Procedure Consistency metrics. Furthermore, CookingDiffusion demonstrates the ability to manipulate ingredients and cooking methods in a recipe. We will make our code, models, and dataset publicly accessible.