# Improving Dietary Assessment Via Integrated Hierarchy Food Classification

arXiv: https://arxiv.org/abs/2109.02736 · HTML: https://mlsanity.com/paper/2109.02736/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [Visual Aware Hierarchy Based Food Recognition](https://mlsanity.com/paper/2012.03368/) (2020)
- [Multi-Task Image-Based Dietary Assessment for Food Recognition and Portion Size Estimation](https://mlsanity.com/paper/2004.13188/) (2020)
- [An End-to-End Food Image Analysis System](https://mlsanity.com/paper/2102.00645/) (2021)
- [Towards Learning Food Portion From Monocular Images With Cross-Domain Feature Adaptation](https://mlsanity.com/paper/2103.07562/) (2021)
- [DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment](https://mlsanity.com/paper/1606.05675/) (2016)
- [Simultaneous Food Localization and Recognition](https://mlsanity.com/paper/1604.07953/) (2016)
- [FoodNet: Recognizing Foods Using Ensemble of Deep Networks](https://mlsanity.com/paper/1709.09429/) (2017)

## Built on by (most cited first, 15 of 15)

- [Long-Tailed Continual Learning For Visual Food Recognition](https://mlsanity.com/paper/2307.00183/) (2023)
- [Long-tailed Food Classification](https://mlsanity.com/paper/2210.14748/) (2022)
- [Online Class-Incremental Learning For Real-World Food Image Classification](https://mlsanity.com/paper/2301.05246/) (2023)
- [An End-to-end Food Portion Estimation Framework Based on Shape Reconstruction from Monocular Image](https://mlsanity.com/paper/2308.01810/) (2023)
- [Food Portion Estimation via 3D Object Scaling](https://mlsanity.com/paper/2404.12257/) (2024)
- [MetaFood CVPR 2024 Challenge on Physically Informed 3D Food Reconstruction: Methods and Results](https://mlsanity.com/paper/2407.09285/) (2024)
- [Simulating Personal Food Consumption Patterns using a Modified Markov Chain](https://mlsanity.com/paper/2208.06709/) (2022)
- [An Improved Encoder-Decoder Framework for Food Energy Estimation](https://mlsanity.com/paper/2309.00468/) (2023)
- [Diffusion Model with Clustering-based Conditioning for Food Image Generation](https://mlsanity.com/paper/2309.00199/) (2023)
- [Muti-Stage Hierarchical Food Classification](https://mlsanity.com/paper/2309.01075/) (2023)
- [Personalized Food Image Classification: Benchmark Datasets and New Baseline](https://mlsanity.com/paper/2309.08744/) (2023)
- [Conditional Synthetic Food Image Generation](https://mlsanity.com/paper/2303.09005/) (2023)
- [Comprehensive Evaluation of Large Multimodal Models for Nutrition Analysis: A New Benchmark Enriched with Contextual Metadata](https://mlsanity.com/paper/2507.07048/) (2025)
- [Learning to Classify New Foods Incrementally Via Compressed Exemplars](https://mlsanity.com/paper/2404.07507/) (2024)
- [FMiFood: Multi-modal Contrastive Learning for Food Image Classification](https://mlsanity.com/paper/2408.03922/) (2024)

## Abstract (arXiv)

Image-based dietary assessment refers to the process of determining what someone eats and how much energy and nutrients are consumed from visual data. Food classification is the first and most crucial step. Existing methods focus on improving accuracy measured by the rate of correct classification based on visual information alone, which is very challenging due to the high complexity and inter-class similarity of foods. Further, accuracy in food classification is conceptual as description of a food can always be improved. In this work, we introduce a new food classification framework to improve the quality of predictions by integrating the information from multiple domains while maintaining the classification accuracy. We apply a multi-task network based on a hierarchical structure that uses both visual and nutrition domain specific information to cluster similar foods. Our method is validated on the modified VIPER-FoodNet (VFN) food image dataset by including associated energy and nutrient information. We achieve comparable classification accuracy with existing methods that use visual information only, but with less error in terms of energy and nutrient values for the wrong predictions.