# ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation

arXiv: https://arxiv.org/abs/2409.12010 · HTML: https://mlsanity.com/paper/2409.12010/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Generating Images with Multimodal Language Models](https://mlsanity.com/paper/2305.17216/) (2023)
- [Inverse Cooking: Recipe Generation from Food Images](https://mlsanity.com/paper/1812.06164/) (2018)
- [Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Images](https://mlsanity.com/paper/1905.01273/) (2019)
- [RecipeGPT: Generative Pre-training Based Cooking Recipe Generation and Evaluation System](https://mlsanity.com/paper/2003.02498/) (2020)
- [How to make a pizza: Learning a compositional layer-based GAN model](https://mlsanity.com/paper/1906.02839/) (2019)
- [Learning Structural Representations for Recipe Generation and Food Retrieval](https://mlsanity.com/paper/2110.01209/) (2021)
- [CookGAN: Meal Image Synthesis from Ingredients](https://mlsanity.com/paper/2002.11493/) (2020)
- [RecipeRec: A Heterogeneous Graph Learning Model for Recipe Recommendation](https://mlsanity.com/paper/2205.14005/) (2022)
- [Recipe2Vec: Multi-modal Recipe Representation Learning with Graph Neural Networks](https://mlsanity.com/paper/2205.12396/) (2022)

## Built on by (most cited first, 11 of 11)

- [AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering](https://mlsanity.com/paper/2510.05445/) (2025)
- [EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answering](https://mlsanity.com/paper/2604.05149/) (2026)
- [Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization](https://mlsanity.com/paper/2605.21801/) (2026)
- [Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generation](https://mlsanity.com/paper/2602.15862/) (2026)
- [SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment](https://mlsanity.com/paper/2606.11512/) (2026)
- [CrochetBench: Can Vision-Language Models Move from Describing to Doing in Crochet Domain?](https://mlsanity.com/paper/2511.09483/) (2025)
- [ChemHGNN: A Hierarchical Hypergraph Neural Network for Reaction Virtual Screening and Discovery](https://mlsanity.com/paper/2506.11041/) (2025)
- [Food Image Generation on Multi-Noun Categories](https://mlsanity.com/paper/2512.09095/) (2025)
- [FoodLogAthl-218: Constructing a Real-World Food Image Dataset Using Dietary Management Applications](https://mlsanity.com/paper/2512.14574/) (2025)
- [Training-Free Text-to-Image Compositional Food Generation via Prompt Grafting](https://mlsanity.com/paper/2601.17666/) (2026)
- [Quantifying Hidden Salt for Precision Healthcare: Sodium Assessment via Joint-Factor Retrieval and Chain-of-Thought Inference](https://mlsanity.com/paper/2609.22171/) (2026)

## Abstract (arXiv)

Significant work has been conducted in the domain of food computing, yet these studies typically focus on single tasks such as t2t (instruction generation from food titles and ingredients), i2t (recipe generation from food images), or t2i (food image generation from recipes). None of these approaches integrate all modalities simultaneously. To address this gap, we introduce a novel food computing foundation model that achieves true multimodality, encompassing tasks such as t2t, t2i, i2t, it2t, and t2ti. By leveraging large language models (LLMs) and pre-trained image encoder and decoder models, our model can perform a diverse array of food computing-related tasks, including food understanding, food recognition, recipe generation, and food image generation. Compared to previous models, our foundation model demonstrates a significantly broader range of capabilities and exhibits superior performance, particularly in food image generation and recipe generation tasks. We open-sourced ChefFusion at GitHub.