# Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis arXiv: https://arxiv.org/abs/2312.08592 ยท HTML: https://mlsanity.com/paper/2312.08592/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 20 of its 48 references that are in the map. - First posted on arXiv 14 Dec 2023. ## Builds on (most cited first, 20 of 20) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023) - [Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges](https://mlsanity.com/paper/2311.03287/) (2023) - [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018) - [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023) - [Large Scale Visual Food Recognition](https://mlsanity.com/paper/2103.16107/) (2021) - [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023) - [Inverse Cooking: Recipe Generation from Food Images](https://mlsanity.com/paper/1812.06164/) (2018) - [A Large-Scale Benchmark for Food Image Segmentation](https://mlsanity.com/paper/2105.05409/) (2021) - [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021) - [VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligence](https://mlsanity.com/paper/2310.04992/) (2023) - [Structure-Aware Generation Network for Recipe Generation from Images](https://mlsanity.com/paper/2009.00944/) (2020) - [Learning Structural Representations for Recipe Generation and Food Retrieval](https://mlsanity.com/paper/2110.01209/) (2021) - [Mining Discriminative Food Regions for Accurate Food Recognition](https://mlsanity.com/paper/2207.03692/) (2022) - [A Data Driven End-to-end Approach for In-the-wild Monitoring of Eating Behavior Using Smartwatches](https://mlsanity.com/paper/2010.07051/) (2020) - [Assessing the Accuracy of a Wrist Motion Tracking Method for Counting Bites across Demographic and Food Variables](https://mlsanity.com/paper/1806.05352/) (2018) - [An Artificial Intelligence-Based System to Assess Nutrient Intake for Hospitalised Patients](https://mlsanity.com/paper/2003.08273/) (2020) - [Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring](https://mlsanity.com/paper/2107.00372/) (2021) - [Partially Supervised Multi-Task Network for Single-View Dietary Assessment](https://mlsanity.com/paper/2008.00818/) (2020) - [Clustering Egocentric Images in Passive Dietary Monitoring with Self-Supervised Learning](https://mlsanity.com/paper/2208.12160/) (2022) ## Built on by (most cited first, 10 of 10) - [MEDCO: Medical Education Copilots Based on A Multi-Agent Framework](https://mlsanity.com/paper/2408.12496/) (2024) - [An LLM-enabled Multi-Agent Autonomous Mechatronics Design Framework](https://mlsanity.com/paper/2504.14681/) (2025) - [Comprehensive Evaluation of Large Multimodal Models for Nutrition Analysis: A New Benchmark Enriched with Contextual Metadata](https://mlsanity.com/paper/2507.07048/) (2025) - [Artificial Intelligence for Biomedical Video Generation](https://mlsanity.com/paper/2411.07619/) (2024) - [A Closed-Loop Multi-Agent System Driven by LLMs for Meal-Level Personalized Nutrition Management](https://mlsanity.com/paper/2601.04491/) (2026) - [RHealthTwin: Towards Responsible and Multimodal Digital Twins for Personalized Well-being](https://mlsanity.com/paper/2506.08486/) (2025) - [Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Cameras](https://mlsanity.com/paper/2510.09230/) (2025) - [NutriMLLM: Multimodal Large Language Models for Dietary Micronutrient Analysis](https://mlsanity.com/paper/2606.08948/) (2026) - [Informatics for Food Processing](https://mlsanity.com/paper/2505.17087/) (2025) - [MultiFoodhat: A potential new paradigm for intelligent food quality inspection](https://mlsanity.com/paper/2510.13889/) (2025) ## Abstract (arXiv) Conventional approaches to dietary assessment are primarily grounded in self-reporting methods or structured interviews conducted under the supervision of dietitians. These methods, however, are often subjective, potentially inaccurate, and time-intensive. Although artificial intelligence (AI)-based solutions have been devised to automate the dietary assessment process, these prior AI methodologies encounter challenges in their ability to generalize across a diverse range of food types, dietary behaviors, and cultural contexts. This results in AI applications in the dietary field that possess a narrow specialization and limited accuracy. Recently, the emergence of multimodal foundation models such as GPT-4V powering the latest ChatGPT has exhibited transformative potential across a wide range of tasks (e.g., Scene understanding and image captioning) in numerous research domains. These models have demonstrated remarkable generalist intelligence and accuracy, capable of processing various data modalities. In this study, we explore the application of multimodal ChatGPT within the realm of dietary assessment. Our findings reveal that GPT-4V excels in food detection under challenging conditions with accuracy up to 87.5% without any fine-tuning or adaptation using food-specific datasets. By guiding the model with specific language prompts (e.g., African cuisine), it shifts from recognizing common staples like rice and bread to accurately identifying regional dishes like banku and ugali. Another GPT-4V's standout feature is its contextual awareness. GPT-4V can leverage surrounding objects as scale references to deduce the portion sizes of food items, further enhancing its accuracy in translating food weight into nutritional content. This alignment with the USDA National Nutrient Database underscores GPT-4V's potential to advance nutritional science and dietary assessment techniques.