# Structure-Aware Generation Network for Recipe Generation from Images

arXiv: https://arxiv.org/abs/2009.00944 · HTML: https://mlsanity.com/paper/2009.00944/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Sep 2020.

## Built on by (most cited first, 19 of 19)

- [Large Scale Visual Food Recognition](https://mlsanity.com/paper/2103.16107/) (2021)
- [A Large-Scale Benchmark for Food Image Segmentation](https://mlsanity.com/paper/2105.05409/) (2021)
- [FoodLMM: A Versatile Food Assistant using Large Multi-modal Model](https://mlsanity.com/paper/2312.14991/) (2023)
- [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021)
- [Cycle-Consistent Inverse GAN for Text-to-Image Synthesis](https://mlsanity.com/paper/2108.01361/) (2021)
- [FIRE: Food Image to REcipe generation](https://mlsanity.com/paper/2308.14391/) (2023)
- [Learning Structural Representations for Recipe Generation and Food Retrieval](https://mlsanity.com/paper/2110.01209/) (2021)
- [Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis](https://mlsanity.com/paper/2312.08592/) (2023)
- [Applications of knowledge graphs for food science and industry](https://mlsanity.com/paper/2107.05869/) (2021)
- [Learning to Substitute Ingredients in Recipes](https://mlsanity.com/paper/2302.07960/) (2023)
- [Ratatouille: A tool for Novel Recipe Generation](https://mlsanity.com/paper/2206.08267/) (2022)
- [RecipeMind: Guiding Ingredient Choices from Food Pairing to Recipe Completion using Cascaded Set Transformer](https://mlsanity.com/paper/2210.10628/) (2022)
- [Monte Carlo Tree Search for Recipe Generation using GPT-2](https://mlsanity.com/paper/2401.05199/) (2024)
- [Retrieval Augmented Recipe Generation](https://mlsanity.com/paper/2411.08715/) (2024)
- [Recipe Generation from Unsegmented Cooking Videos](https://mlsanity.com/paper/2209.10134/) (2022)
- [A Closed-Loop Multi-Agent System Driven by LLMs for Meal-Level Personalized Nutrition Management](https://mlsanity.com/paper/2601.04491/) (2026)
- [MALM: Mask Augmentation based Local Matching for Food-Recipe Retrieval](https://mlsanity.com/paper/2305.11327/) (2023)
- [RecipeGen: A Benchmark for Real-World Recipe Image Generation](https://mlsanity.com/paper/2503.05228/) (2025)
- [RecipeGen: A Step-Aligned Multimodal Benchmark for Real-World Recipe Generation](https://mlsanity.com/paper/2506.06733/) (2025)

## Abstract (arXiv)

Sharing food has become very popular with the development of social media. For many real-world applications, people are keen to know the underlying recipes of a food item. In this paper, we are interested in automatically generating cooking instructions for food. We investigate an open research task of generating cooking instructions based on only food images and ingredients, which is similar to the image captioning task. However, compared with image captioning datasets, the target recipes are long-length paragraphs and do not have annotations on structure information. To address the above limitations, we propose a novel framework of Structure-aware Generation Network (SGN) to tackle the food recipe generation task. Our approach brings together several novel ideas in a systematic framework: (1) exploiting an unsupervised learning approach to obtain the sentence-level tree structure labels before training; (2) generating trees of target recipes from images with the supervision of tree structure labels learned from (1); and (3) integrating the inferred tree structures with the recipe generation procedure. Our proposed model can produce high-quality and coherent recipes, and achieve the state-of-the-art performance on the benchmark Recipe1M dataset.