# FIRE: Food Image to REcipe generation

arXiv: https://arxiv.org/abs/2308.14391 · HTML: https://mlsanity.com/paper/2308.14391/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 75 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Language Models of Code are Few-Shot Commonsense Learners](https://mlsanity.com/paper/2210.07128/) (2022)
- [Detecting Hallucinated Content in Conditional Neural Sequence Generation](https://mlsanity.com/paper/2011.02593/) (2020)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [A Survey on Food Computing](https://mlsanity.com/paper/1808.07202/) (2018)
- [Inverse Cooking: Recipe Generation from Food Images](https://mlsanity.com/paper/1812.06164/) (2018)
- [CSKG: The CommonSense Knowledge Graph](https://mlsanity.com/paper/2012.11490/) (2020)
- [Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval](https://mlsanity.com/paper/2304.03391/) (2023)
- [Structure-Aware Generation Network for Recipe Generation from Images](https://mlsanity.com/paper/2009.00944/) (2020)
- [Wide-Slice Residual Networks for Food Recognition](https://mlsanity.com/paper/1612.06543/) (2016)
- [Learning Structural Representations for Recipe Generation and Food Retrieval](https://mlsanity.com/paper/2110.01209/) (2021)
- [Learning Program Representations for Food Images and Cooking Recipes](https://mlsanity.com/paper/2203.16071/) (2022)

## Built on by (most cited first, 18 of 18)

- [MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs](https://mlsanity.com/paper/2502.17422/) (2025)
- [Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models](https://mlsanity.com/paper/2502.11028/) (2025)
- [FoodLMM: A Versatile Food Assistant using Large Multi-modal Model](https://mlsanity.com/paper/2312.14991/) (2023)
- [LLaVA-Chef: A Multi-modal Generative Model for Food Recipes](https://mlsanity.com/paper/2408.16889/) (2024)
- [RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models](https://mlsanity.com/paper/2407.12730/) (2024)
- [Advancing Food Nutrition Estimation via Visual-Ingredient Feature Fusion](https://mlsanity.com/paper/2505.08747/) (2025)
- [CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion](https://mlsanity.com/paper/2501.09042/) (2025)
- [Coherent Zero-Shot Visual Instruction Generation](https://mlsanity.com/paper/2406.04337/) (2024)
- [Fine-tuning Language Models for Recipe Generation: A Comparative Analysis and Benchmark Study](https://mlsanity.com/paper/2502.02028/) (2025)
- [Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generation](https://mlsanity.com/paper/2602.15862/) (2026)
- [FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation](https://mlsanity.com/paper/2409.18459/) (2024)
- [KERL: Knowledge-Enhanced Personalized Recipe Recommendation using Large Language Models](https://mlsanity.com/paper/2505.14629/) (2025)
- [SFOOD: A Multimodal Benchmark for Comprehensive Food Attribute Analysis Beyond RGB with Spectral Insights](https://mlsanity.com/paper/2507.04412/) (2025)
- [RecipeGen: A Benchmark for Real-World Recipe Image Generation](https://mlsanity.com/paper/2503.05228/) (2025)
- [Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learning](https://mlsanity.com/paper/2511.13351/) (2025)
- [Efficient Test-Time Retrieval Augmented Generation](https://mlsanity.com/paper/2511.01059/) (2025)
- [Towards Unbiased Cross-Modal Representation Learning for Food Image-to-Recipe Retrieval](https://mlsanity.com/paper/2511.15201/) (2025)
- [Food-R1: A Unified Multi-Task Food Vision-Language Model with Reinforcement Learning](https://mlsanity.com/paper/2606.04986/) (2026)

## Abstract (arXiv)

Food computing has emerged as a prominent multidisciplinary field of research in recent years. An ambitious goal of food computing is to develop end-to-end intelligent systems capable of autonomously producing recipe information for a food image. Current image-to-recipe methods are retrieval-based and their success depends heavily on the dataset size and diversity, as well as the quality of learned embeddings. Meanwhile, the emergence of powerful attention-based vision and language models presents a promising avenue for accurate and generalizable recipe generation, which has yet to be extensively explored. This paper proposes FIRE, a novel multimodal methodology tailored to recipe generation in the food computing domain, which generates the food title, ingredients, and cooking instructions based on input food images. FIRE leverages the BLIP model to generate titles, utilizes a Vision Transformer with a decoder for ingredient extraction, and employs the T5 model to generate recipes incorporating titles and ingredients as inputs. We showcase two practical applications that can benefit from integrating FIRE with large language model prompting: recipe customization to fit recipes to user preferences and recipe-to-code transformation to enable automated cooking processes. Our experimental findings validate the efficacy of our proposed approach, underscoring its potential for future advancements and widespread adoption in food computing.