# Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food

arXiv: https://arxiv.org/abs/2103.03375 · HTML: https://mlsanity.com/paper/2103.03375/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 4 Mar 2021.

## Builds on (most cited first, 7 of 7)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [Simultaneous Food Localization and Recognition](https://mlsanity.com/paper/1604.07953/) (2016)
- [MUSEFood: Multi-sensor-based Food Volume Estimation on Smartphones](https://mlsanity.com/paper/1903.07437/) (2019)
- [Computer vision-based food calorie estimation: dataset, method, and experiment](https://mlsanity.com/paper/1705.07632/) (2017)
- [Multi-Task Learning for Calorie Prediction on a Novel Large-Scale Recipe Dataset Enriched with Nutritional Information](https://mlsanity.com/paper/2011.01082/) (2020)

## Built on by (most cited first, 30 of 58)

- [Large Scale Visual Food Recognition](https://mlsanity.com/paper/2103.16107/) (2021)
- [A Large-Scale Benchmark for Food Image Segmentation](https://mlsanity.com/paper/2105.05409/) (2021)
- [FoodLMM: A Versatile Food Assistant using Large Multi-modal Model](https://mlsanity.com/paper/2312.14991/) (2023)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [NutriBench: A Dataset for Evaluating Large Language Models on Nutrition Estimation from Meal Descriptions](https://mlsanity.com/paper/2407.12843/) (2024)
- [An End-to-end Food Portion Estimation Framework Based on Shape Reconstruction from Monocular Image](https://mlsanity.com/paper/2308.01810/) (2023)
- [On Domain-Adaptive Post-Training for Multimodal Large Language Models](https://mlsanity.com/paper/2411.19930/) (2024)
- [Food Portion Estimation via 3D Object Scaling](https://mlsanity.com/paper/2404.12257/) (2024)
- [RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models](https://mlsanity.com/paper/2407.12730/) (2024)
- [Image Based Food Energy Estimation With Depth Domain Adaptation](https://mlsanity.com/paper/2208.12153/) (2022)
- [DPF-Nutrition: Food Nutrition Estimation via Depth Prediction and Fusion](https://mlsanity.com/paper/2310.11702/) (2023)
- [FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination](https://mlsanity.com/paper/2406.10261/) (2024)
- [A review on vision-based analysis for automatic dietary assessment](https://mlsanity.com/paper/2108.02947/) (2021)
- [GrainSpace: A Large-scale Dataset for Fine-grained and Domain-adaptive Recognition of Cereal Grains](https://mlsanity.com/paper/2203.05306/) (2022)
- [From Canteen Food to Daily Meals: Generalizing Food Recognition to More Practical Scenarios](https://mlsanity.com/paper/2403.07403/) (2024)
- [MetaFood CVPR 2024 Challenge on Physically Informed 3D Food Reconstruction: Methods and Results](https://mlsanity.com/paper/2407.09285/) (2024)
- [MFP3D: Monocular Food Portion Estimation Leveraging 3D Point Clouds](https://mlsanity.com/paper/2411.10492/) (2024)
- [Applications of knowledge graphs for food science and industry](https://mlsanity.com/paper/2107.05869/) (2021)
- [Are Vision-Language Models Ready for Dietary Assessment? Exploring the Next Frontier in AI-Powered Food Image Recognition](https://mlsanity.com/paper/2504.06925/) (2025)
- [GLEN-Bench: A Graph-Language based Benchmark for Nutritional Health](https://mlsanity.com/paper/2601.18106/) (2026)
- [NutritionVerse-3D: A 3D Food Model Dataset for Nutritional Intake Estimation](https://mlsanity.com/paper/2304.05619/) (2023)
- [Transferring Knowledge for Food Image Segmentation using Transformers and Convolutions](https://mlsanity.com/paper/2306.09203/) (2023)
- [Identifying the Defective: Detecting Damaged Grains for Cereal Appearance Inspection](https://mlsanity.com/paper/2311.11901/) (2023)
- [OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation](https://mlsanity.com/paper/2404.01409/) (2024)
- [VolETA: One- and Few-shot Food Volume Estimation](https://mlsanity.com/paper/2407.01717/) (2024)
- [Advancing Food Nutrition Estimation via Visual-Ingredient Feature Fusion](https://mlsanity.com/paper/2505.08747/) (2025)
- [Towards the Creation of a Nutrition and Food Group Based Image Database](https://mlsanity.com/paper/2206.02086/) (2022)
- [Nutrition Estimation for Dietary Management: A Transformer Approach with Depth Sensing](https://mlsanity.com/paper/2406.01938/) (2024)
- [FoodFusion: A Latent Diffusion Model for Realistic Food Image Generation](https://mlsanity.com/paper/2312.03540/) (2023)
- [FoodMem: Near Real-time and Precise Food Video Segmentation](https://mlsanity.com/paper/2407.12121/) (2024)

## Abstract (arXiv)

Understanding the nutritional content of food from visual data is a challenging computer vision problem, with the potential to have a positive and widespread impact on public health. Studies in this area are limited to existing datasets in the field that lack sufficient diversity or labels required for training models with nutritional understanding capability. We introduce Nutrition5k, a novel dataset of 5k diverse, real world food dishes with corresponding video streams, depth images, component weights, and high accuracy nutritional content annotation. We demonstrate the potential of this dataset by training a computer vision algorithm capable of predicting the caloric and macronutrient values of a complex, real world dish at an accuracy that outperforms professional nutritionists. Further we present a baseline for incorporating depth sensor data to improve nutrition predictions. We will publicly release Nutrition5k in the hope that it will accelerate innovation in the space of nutritional understanding.