# LLaVA-Chef: A Multi-modal Generative Model for Food Recipes

arXiv: https://arxiv.org/abs/2408.16889 · HTML: https://mlsanity.com/paper/2408.16889/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Inverse Cooking: Recipe Generation from Food Images](https://mlsanity.com/paper/1812.06164/) (2018)
- [Online Continual Learning For Visual Food Classification](https://mlsanity.com/paper/2108.06781/) (2021)
- [RecipeGPT: Generative Pre-training Based Cooking Recipe Generation and Evaluation System](https://mlsanity.com/paper/2003.02498/) (2020)
- [FIRE: Food Image to REcipe generation](https://mlsanity.com/paper/2308.14391/) (2023)
- [Learning to Substitute Ingredients in Recipes](https://mlsanity.com/paper/2302.07960/) (2023)
- [A Big Data Approach to Computational Creativity](https://mlsanity.com/paper/1311.1213/) (2013)

## Built on by (most cited first, 14 of 14)

- [Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding](https://mlsanity.com/paper/2505.16990/) (2025)
- [On Domain-Adaptive Post-Training for Multimodal Large Language Models](https://mlsanity.com/paper/2411.19930/) (2024)
- [Are Vision-Language Models Ready for Dietary Assessment? Exploring the Next Frontier in AI-Powered Food Image Recognition](https://mlsanity.com/paper/2504.06925/) (2025)
- [Fine-tuning Language Models for Recipe Generation: A Comparative Analysis and Benchmark Study](https://mlsanity.com/paper/2502.02028/) (2025)
- [The Effectiveness of Large Language Models in Transforming Unstructured Text to Standardized Formats](https://mlsanity.com/paper/2503.02650/) (2025)
- [KERL: Knowledge-Enhanced Personalized Recipe Recommendation using Large Language Models](https://mlsanity.com/paper/2505.14629/) (2025)
- [SFOOD: A Multimodal Benchmark for Comprehensive Food Attribute Analysis Beyond RGB with Spectral Insights](https://mlsanity.com/paper/2507.04412/) (2025)
- [VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models](https://mlsanity.com/paper/2510.13808/) (2025)
- [Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA](https://mlsanity.com/paper/2605.14928/) (2026)
- [SpiderGen: Towards Procedure Generation For Carbon Life Cycle Assessments with Generative AI](https://mlsanity.com/paper/2511.10684/) (2025)
- [Food-R1: A Unified Multi-Task Food Vision-Language Model with Reinforcement Learning](https://mlsanity.com/paper/2606.04986/) (2026)
- [Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention](https://mlsanity.com/paper/2607.23917/) (2026)
- [From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding](https://mlsanity.com/paper/2609.00948/) (2026)
- [Quantifying Hidden Salt for Precision Healthcare: Sodium Assessment via Joint-Factor Retrieval and Chain-of-Thought Inference](https://mlsanity.com/paper/2609.22171/) (2026)

## Abstract (arXiv)

In the rapidly evolving landscape of online recipe sharing within a globalized context, there has been a notable surge in research towards comprehending and generating food recipes. Recent advancements in large language models (LLMs) like GPT-2 and LLaVA have paved the way for Natural Language Processing (NLP) approaches to delve deeper into various facets of food-related tasks, encompassing ingredient recognition and comprehensive recipe generation. Despite impressive performance and multi-modal adaptability of LLMs, domain-specific training remains paramount for their effective application. This work evaluates existing LLMs for recipe generation and proposes LLaVA-Chef, a novel model trained on a curated dataset of diverse recipe prompts in a multi-stage approach. First, we refine the mapping of visual food image embeddings to the language space. Second, we adapt LLaVA to the food domain by fine-tuning it on relevant recipe data. Third, we utilize diverse prompts to enhance the model's recipe comprehension. Finally, we improve the linguistic quality of generated recipes by penalizing the model with a custom loss function. LLaVA-Chef demonstrates impressive improvements over pretrained LLMs and prior works. A detailed qualitative analysis reveals that LLaVA-Chef generates more detailed recipes with precise ingredient mentions, compared to existing approaches.