# Inverse Cooking: Recipe Generation from Food Images

arXiv: https://arxiv.org/abs/1812.06164 · HTML: https://mlsanity.com/paper/1812.06164/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2018.

## Built on by (most cited first, 30 of 47)

- [Large Scale Visual Food Recognition](https://mlsanity.com/paper/2103.16107/) (2021)
- [A Large-Scale Benchmark for Food Image Segmentation](https://mlsanity.com/paper/2105.05409/) (2021)
- [FoodLMM: A Versatile Food Assistant using Large Multi-modal Model](https://mlsanity.com/paper/2312.14991/) (2023)
- [Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation](https://mlsanity.com/paper/2109.05743/) (2021)
- [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021)
- [Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning](https://mlsanity.com/paper/2206.01843/) (2022)
- [FIRE: Food Image to REcipe generation](https://mlsanity.com/paper/2308.14391/) (2023)
- [Learning Structural Representations for Recipe Generation and Food Retrieval](https://mlsanity.com/paper/2110.01209/) (2021)
- [Learning Program Representations for Food Images and Cooking Recipes](https://mlsanity.com/paper/2203.16071/) (2022)
- [SHARE: a System for Hierarchical Assistive Recipe Editing](https://mlsanity.com/paper/2105.08185/) (2021)
- [LLaVA-Chef: A Multi-modal Generative Model for Food Recipes](https://mlsanity.com/paper/2408.16889/) (2024)
- [RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models](https://mlsanity.com/paper/2407.12730/) (2024)
- [ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation](https://mlsanity.com/paper/2409.12010/) (2024)
- [Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions](https://mlsanity.com/paper/2106.03158/) (2021)
- [Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis](https://mlsanity.com/paper/2312.08592/) (2023)
- [FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination](https://mlsanity.com/paper/2406.10261/) (2024)
- [A Comprehensive Survey of Image-Based Food Recognition and Volume Estimation Methods for Dietary Assessment](https://mlsanity.com/paper/2106.11776/) (2021)
- [Learning to Substitute Ingredients in Recipes](https://mlsanity.com/paper/2302.07960/) (2023)
- [Plug-and-Play Recipe Generation with Content Planning](https://mlsanity.com/paper/2212.05093/) (2022)
- [Ratatouille: A tool for Novel Recipe Generation](https://mlsanity.com/paper/2206.08267/) (2022)
- [RecipeMind: Guiding Ingredient Choices from Food Pairing to Recipe Completion using Cascaded Set Transformer](https://mlsanity.com/paper/2210.10628/) (2022)
- [Non-Sequential Graph Script Induction via Multimedia Grounding](https://mlsanity.com/paper/2305.17542/) (2023)
- [CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion](https://mlsanity.com/paper/2501.09042/) (2025)
- [Coherent Zero-Shot Visual Instruction Generation](https://mlsanity.com/paper/2406.04337/) (2024)
- [Retrieval Augmented Recipe Generation](https://mlsanity.com/paper/2411.08715/) (2024)
- [Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generation](https://mlsanity.com/paper/2602.15862/) (2026)
- [Food Recipe Recommendation Based on Ingredients Detection Using Deep Learning](https://mlsanity.com/paper/2203.06721/) (2022)
- [Assistive Recipe Editing through Critiquing](https://mlsanity.com/paper/2205.02454/) (2022)
- [Recipe Generation from Unsegmented Cooking Videos](https://mlsanity.com/paper/2209.10134/) (2022)
- [Human in the loop approaches in multi-modal conversational task guidance system development](https://mlsanity.com/paper/2211.01824/) (2022)

## Abstract (arXiv)

People enjoy food photography because they appreciate food. Behind each meal there is a story described in a complex recipe and, unfortunately, by simply looking at a food image we do not have access to its preparation process. Therefore, in this paper we introduce an inverse cooking system that recreates cooking recipes given food images. Our system predicts ingredients as sets by means of a novel architecture, modeling their dependencies without imposing any order, and then generates cooking instructions by attending to both image and its inferred ingredients simultaneously. We extensively evaluate the whole system on the large-scale Recipe1M dataset and show that (1) we improve performance w.r.t. previous baselines for ingredient prediction; (2) we are able to obtain high quality recipes by leveraging both image and ingredients; (3) our system is able to produce more compelling recipes than retrieval-based approaches according to human judgment. We make code and models publicly available.