Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generation
On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment
Abstract (arXiv)
Recent advances in Multimodal Large Language Models (MLMMs) have enabled recipe generation from food images, yet outputs often contain semantically incorrect actions or ingredients despite high lexical scores (e.g., BLEU, ROUGE). To address this gap, we propose a semantically grounded framework that predicts and validates actions and ingredients as internal context for instruction generation. Our two-stage pipeline combines supervised fine-tuning (SFT) with reinforcement fine-tuning (RFT): SFT builds foundational accuracy using an Action-Reasoning dataset and ingredient corpus, while RFT employs frequency-aware rewards to improve long-tail action prediction and ingredient generalization. A Semantic Confidence Scoring and Rectification (SCSR) module further filters and corrects predictions. Experiments on Recipe1M show state-of-the-art performance and markedly improved semantic fidelity.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 22 references that are in the map. First posted on arXiv 26 Jan 2026.
Builds on
20 of 22 references.
References · 20
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024GPT-4o System Cardcited by 4,467
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
- 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
- 2018Inverse Cooking: Recipe Generation from Food Imagescited by 47
8 more
- 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
- 2023FoodLMM: A Versatile Food Assistant using Large Multi-modal Modelcited by 32
- 2023Retrieval-augmented Image Captioningcited by 28
- 2023FIRE: Food Image to REcipe generationcited by 18
- 2024ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generationcited by 11
- 2024From Canteen Food to Daily Meals: Generalizing Food Recognition to More Practical Scenarioscited by 9
- 2024Navigating Weight Prediction with Diet Diarycited by 7
- 2024Retrieval Augmented Recipe Generationcited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Directed Domain Fine-Tuning: Tailoring Separate Modalities for Specific Training Tasks
- 2024Large Language Models for Ingredient Substitution in Food Recipes using Supervised Fine-tuning and Direct Preference Optimization
- 2025CookAnything: A Framework for Flexible and Consistent Multi-Step Recipe Image Generation
- 2025Efficient Prompt Tuning for Hierarchical Ingredient Recognition
- 2026CGU-ILALab at FoodBench-QA 2026: Comparing Traditional and LLM-based Approaches for Recipe Nutrient Estimation
- 2022Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows
- 2024PizzaCommonSense: Learning to Model Commonsense Reasoning about Intermediate Steps in Cooking Recipes
- 2026Do Language Models Know Their Own Constraints?