# An End-to-End Food Image Analysis System

arXiv: https://arxiv.org/abs/2102.00645 · HTML: https://mlsanity.com/paper/2102.00645/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 1 Feb 2021.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Incremental Learning In Online Scenario](https://mlsanity.com/paper/2003.13191/) (2020)
- [Visual Aware Hierarchy Based Food Recognition](https://mlsanity.com/paper/2012.03368/) (2020)
- [Multi-Task Image-Based Dietary Assessment for Food Recognition and Portion Size Estimation](https://mlsanity.com/paper/2004.13188/) (2020)
- [DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment](https://mlsanity.com/paper/1606.05675/) (2016)
- [Single-View Food Portion Estimation: Learning Image-to-Energy Mappings Using Generative Adversarial Networks](https://mlsanity.com/paper/1802.09670/) (2018)
- [Simultaneous Food Localization and Recognition](https://mlsanity.com/paper/1604.07953/) (2016)
- [FoodNet: Recognizing Foods Using Ensemble of Deep Networks](https://mlsanity.com/paper/1709.09429/) (2017)

## Built on by (most cited first, 21 of 21)

- [Online Continual Learning For Visual Food Classification](https://mlsanity.com/paper/2108.06781/) (2021)
- [Long-Tailed Continual Learning For Visual Food Recognition](https://mlsanity.com/paper/2307.00183/) (2023)
- [An Integrated System for Mobile Image-Based Dietary Assessment](https://mlsanity.com/paper/2110.01754/) (2021)
- [Long-tailed Food Classification](https://mlsanity.com/paper/2210.14748/) (2022)
- [Improving Dietary Assessment Via Integrated Hierarchy Food Classification](https://mlsanity.com/paper/2109.02736/) (2021)
- [Food Portion Estimation via 3D Object Scaling](https://mlsanity.com/paper/2404.12257/) (2024)
- [Image Based Food Energy Estimation With Depth Domain Adaptation](https://mlsanity.com/paper/2208.12153/) (2022)
- [A review on vision-based analysis for automatic dietary assessment](https://mlsanity.com/paper/2108.02947/) (2021)
- [MetaFood CVPR 2024 Challenge on Physically Informed 3D Food Reconstruction: Methods and Results](https://mlsanity.com/paper/2407.09285/) (2024)
- [MFP3D: Monocular Food Portion Estimation Leveraging 3D Point Clouds](https://mlsanity.com/paper/2411.10492/) (2024)
- [Simulating Personal Food Consumption Patterns using a Modified Markov Chain](https://mlsanity.com/paper/2208.06709/) (2022)
- [An Improved Encoder-Decoder Framework for Food Energy Estimation](https://mlsanity.com/paper/2309.00468/) (2023)
- [Towards the Creation of a Nutrition and Food Group Based Image Database](https://mlsanity.com/paper/2206.02086/) (2022)
- [Diffusion Model with Clustering-based Conditioning for Food Image Generation](https://mlsanity.com/paper/2309.00199/) (2023)
- [Muti-Stage Hierarchical Food Classification](https://mlsanity.com/paper/2309.01075/) (2023)
- [Personalized Food Image Classification: Benchmark Datasets and New Baseline](https://mlsanity.com/paper/2309.08744/) (2023)
- [Conditional Synthetic Food Image Generation](https://mlsanity.com/paper/2303.09005/) (2023)
- [Self-Supervised Visual Representation Learning on Food Images](https://mlsanity.com/paper/2303.09046/) (2023)
- [How Much You Ate? Food Portion Estimation on Spoons](https://mlsanity.com/paper/2405.08717/) (2024)
- [VolE: A Point-cloud Framework for Food 3D Reconstruction and Volume Estimation](https://mlsanity.com/paper/2505.10205/) (2025)
- [V-Nutri: Dish-Level Nutrition Estimation from Egocentric Cooking Videos](https://mlsanity.com/paper/2604.11913/) (2026)

## Abstract (arXiv)

Modern deep learning techniques have enabled advances in image-based dietary assessment such as food recognition and food portion size estimation. Valuable information on the types of foods and the amount consumed are crucial for prevention of many chronic diseases. However, existing methods for automated image-based food analysis are neither end-to-end nor are capable of processing multiple tasks (e.g., recognition and portion estimation) together, making it difficult to apply to real life applications. In this paper, we propose an image-based food analysis framework that integrates food localization, classification and portion size estimation. Our proposed framework is end-to-end, i.e., the input can be an arbitrary food image containing multiple food items and our system can localize each single food item with its corresponding predicted food type and portion size. We also improve the single food portion estimation by consolidating localization results with a food energy distribution map obtained by conditional GAN to generate a four-channel RGB-Distribution image. Our end-to-end framework is evaluated on a real life food image dataset collected from a nutrition feeding study.