mlsanityData to 14 Oct 2026

FoodLMM: A Versatile Food Assistant using Large Multi-modal Model

arXiv 2312.14991 ↗First posted 22 Dec 2023cs.CVcited by 32

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Multi-modal Models (LMMs) have made impressive progress in many vision-language tasks. Nevertheless, the performance of general LMMs in specific domains is still far from satisfactory. This paper proposes FoodLMM, a versatile food assistant based on LMMs with various capabilities, including food recognition, ingredient recognition, recipe generation, nutrition estimation, food segmentation and multi-round conversation. To facilitate FoodLMM to deal with tasks beyond pure text output, we introduce a series of novel task-specific tokens and heads, enabling the model to predict food nutritional values and multiple segmentation masks. We adopt a two-stage training strategy. In the first stage, we utilize multiple public food benchmarks for multi-task learning by leveraging the instruct-following paradigm. In the second stage, we construct a multi-round conversation dataset and a reasoning segmentation dataset to fine-tune the model, enabling it to conduct professional dialogues and generate segmentation masks based on complex reasoning in the food domain. Our fine-tuned FoodLMM achieves state-of-the-art results across several food benchmarks. We will make our code, models and datasets publicly available.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 128 references that are in the map. First posted on arXiv 22 Dec 2023.

Builds on

65 of 128 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Segment Anythingcited by 8,234
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
38 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  2. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  3. 2023BloombergGPT: A Large Language Model for Financecited by 939
  4. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  5. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  7. 2018Do Better ImageNet Models Transfer Better?cited by 565
  8. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  9. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  10. 2020Asymmetric Loss For Multi-Label Classificationcited by 381
  11. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  12. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  13. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  14. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  15. 2019Multi-Label Image Recognition with Graph Convolutional Networkscited by 322
  16. 2016Segmentation from Natural Language Expressionscited by 257
  17. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  18. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  19. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  20. 2020Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasetscited by 122
  21. 2023BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinecited by 100
  22. 2020Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patchescited by 92
  23. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  24. 2021Large Scale Visual Food Recognitioncited by 75
  25. 2023Beyond One-to-One: Rethinking the Referring Image Segmentationcited by 68
  26. 2021Nutrition5k: Towards Automatic Nutritional Understanding of Generic Foodcited by 58
  27. 2019See Better Before Looking Closer: Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classificationcited by 56
  28. 2018Inverse Cooking: Recipe Generation from Food Imagescited by 47
  29. 2020ISIA Food-500: A Dataset for Large-Scale Food Recognition via Stacked Global-Local Attention Networkcited by 47
  30. 2021A Large-Scale Benchmark for Food Image Segmentationcited by 45
  31. 2020RecipeGPT: Generative Pre-training Based Cooking Recipe Generation and Evaluation Systemcited by 34
  32. 2017ChineseFoodNet: A large-scale Image Dataset for Chinese Food Recognitioncited by 21
  33. 2019Food Recommendation: Framework, Existing Solutions and Challengescited by 21
  34. 2020Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanismcited by 20
  35. 2020Deep Semantic Dictionary Learning for Multi-label Image Classificationcited by 20
  36. 2016DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessmentcited by 19
  37. 2020Structure-Aware Generation Network for Recipe Generation from Imagescited by 19
  38. 2023FoodSAM: Any Food Segmentationcited by 19

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 14; 2026: 8.

32 papers.

Sort:

By area:

  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  2. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 57
  3. 2024WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisinescited by 39
  4. 2024On Domain-Adaptive Post-Training for Multimodal Large Language Modelscited by 13
  5. 2024RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Modelscited by 11
  6. 2024FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examinationcited by 10
  7. 2025Are Vision-Language Models Ready for Dietary Assessment? Exploring the Next Frontier in AI-Powered Food Image Recognitioncited by 7
  8. 2025Advancing Food Nutrition Estimation via Visual-Ingredient Feature Fusioncited by 5
  9. 2025CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusioncited by 4
  10. 2024Retrieval Augmented Recipe Generationcited by 3
  11. 2024From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuningcited by 3
  12. 2026Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generationcited by 3
20 more
  1. 2024EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Modelscited by 2
  2. 2024FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generationcited by 2
  3. 2025KERL: Knowledge-Enhanced Personalized Recipe Recommendation using Large Language Modelscited by 2
  4. 2025SFOOD: A Multimodal Benchmark for Comprehensive Food Attribute Analysis Beyond RGB with Spectral Insightscited by 2
  5. 2025RecipeGen: A Benchmark for Real-World Recipe Image Generationcited by 1
  6. 2025Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learningcited by 1
  7. 2025CookAnything: A Framework for Flexible and Consistent Multi-Step Recipe Image Generationcited by 1
  8. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAGcited by 1
  9. 2025RecipeGen: A Step-Aligned Multimodal Benchmark for Real-World Recipe Generationcited by 0
  10. 2025Efficient Test-Time Retrieval Augmented Generationcited by 0
  11. 2025Thought-For-Food: Reasoning Chain Induced Food Visual Question Answeringcited by 0
  12. 2025LLMs-based Augmentation for Domain Adaptation in Long-tailed Food Datasetscited by 0
  13. 2025FoodLogAthl-218: Constructing a Real-World Food Image Dataset Using Dietary Management Applicationscited by 0
  14. 2026VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrievalcited by 0
  15. 2026CogRAG: Tackling Heterogeneous Cognitive Demands in RAG via Stratified Retrieval and Reasoningcited by 0
  16. 2026FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysiscited by 0
  17. 2026Food-R1: A Unified Multi-Task Food Vision-Language Model with Reinforcement Learningcited by 0
  18. 2026ReGraph: Learning to Generate Recipe Graphs from Food Imagescited by 0
  19. 2026RailGen: Improving Railway Intrusion Detection via Agent-Guided Small-Scale Foreign Object Generationcited by 0
  20. 2026Quantifying Hidden Salt for Precision Healthcare: Sodium Assessment via Joint-Factor Retrieval and Chain-of-Thought Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FAM-Bench: A Multimodal Benchmark for Condition-Aware Food-as-Medicine Reasoning
  2. 2025MultiFoodhat: A potential new paradigm for intelligent food quality inspection
  3. 2026OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice
  4. 2023Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models
  5. 2026DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain
  6. 2026CGU-ILALab at FoodBench-QA 2026: Comparing Traditional and LLM-based Approaches for Recipe Nutrient Estimation
  7. 2026SIMMER: Cross-Modal Food Image--Recipe Retrieval via MLLM-Based Embedding
  8. 2022Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval