# NutritionVerse: Empirical Study of Various Dietary Intake Estimation Approaches

arXiv: https://arxiv.org/abs/2309.07704 · HTML: https://mlsanity.com/paper/2309.07704/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [Large Scale Visual Food Recognition](https://mlsanity.com/paper/2103.16107/) (2021)
- [Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food](https://mlsanity.com/paper/2103.03375/) (2021)
- [Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling](https://mlsanity.com/paper/2109.11103/) (2021)
- [FoodX-251: A Dataset for Fine-grained Food Classification](https://mlsanity.com/paper/1907.06167/) (2019)
- [ChineseFoodNet: A large-scale Image Dataset for Chinese Food Recognition](https://mlsanity.com/paper/1705.02743/) (2017)
- [Simultaneous Food Localization and Recognition](https://mlsanity.com/paper/1604.07953/) (2016)
- [Computer vision-based food calorie estimation: dataset, method, and experiment](https://mlsanity.com/paper/1705.07632/) (2017)
- [NutritionVerse-3D: A 3D Food Model Dataset for Nutritional Intake Estimation](https://mlsanity.com/paper/2304.05619/) (2023)
- [When Segmentation is Not Enough: Rectifying Visual-Volume Discordance Through Multisensor Depth-Refined Semantic Segmentation for Food Intake Tracking in Long-Term Care](https://mlsanity.com/paper/1910.11250/) (2019)

## Built on by (most cited first, 2 of 2)

- [NutritionVerse-Direct: Exploring Deep Neural Networks for Multitask Nutrition Prediction from Food Images](https://mlsanity.com/paper/2405.07814/) (2024)
- [Geometry-Enhanced Portion Estimation for Multimodal LLMs](https://mlsanity.com/paper/2607.16514/) (2026)

## Abstract (arXiv)

Accurate dietary intake estimation is critical for informing policies and programs to support healthy eating, as malnutrition has been directly linked to decreased quality of life. However self-reporting methods such as food diaries suffer from substantial bias. Other conventional dietary assessment techniques and emerging alternative approaches such as mobile applications incur high time costs and may necessitate trained personnel. Recent work has focused on using computer vision and machine learning to automatically estimate dietary intake from food images, but the lack of comprehensive datasets with diverse viewpoints, modalities and food annotations hinders the accuracy and realism of such methods. To address this limitation, we introduce NutritionVerse-Synth, the first large-scale dataset of 84,984 photorealistic synthetic 2D food images with associated dietary information and multimodal annotations (including depth images, instance masks, and semantic masks). Additionally, we collect a real image dataset, NutritionVerse-Real, containing 889 images of 251 dishes to evaluate realism. Leveraging these novel datasets, we develop and benchmark NutritionVerse, an empirical study of various dietary intake estimation approaches, including indirect segmentation-based and direct prediction networks. We further fine-tune models pretrained on synthetic data with real images to provide insights into the fusion of synthetic and real data. Finally, we release both datasets (NutritionVerse-Synth, NutritionVerse-Real) on https://www.kaggle.com/nutritionverse/datasets as part of an open initiative to accelerate machine learning for dietary sensing.