# Muti-Stage Hierarchical Food Classification

arXiv: https://arxiv.org/abs/2309.01075 · HTML: https://mlsanity.com/paper/2309.01075/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 52 references that are in the map.
- First posted on arXiv 3 Sep 2023.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Large Scale Visual Food Recognition](https://mlsanity.com/paper/2103.16107/) (2021)
- [Incremental Learning In Online Scenario](https://mlsanity.com/paper/2003.13191/) (2020)
- [ISIA Food-500: A Dataset for Large-Scale Food Recognition via Stacked Global-Local Attention Network](https://mlsanity.com/paper/2008.05655/) (2020)
- [Online Continual Learning For Visual Food Classification](https://mlsanity.com/paper/2108.06781/) (2021)
- [Visual Aware Hierarchy Based Food Recognition](https://mlsanity.com/paper/2012.03368/) (2020)
- [Multi-Task Image-Based Dietary Assessment for Food Recognition and Portion Size Estimation](https://mlsanity.com/paper/2004.13188/) (2020)
- [An End-to-End Food Image Analysis System](https://mlsanity.com/paper/2102.00645/) (2021)
- [Towards Learning Food Portion From Monocular Images With Cross-Domain Feature Adaptation](https://mlsanity.com/paper/2103.07562/) (2021)
- [Long-Tailed Continual Learning For Visual Food Recognition](https://mlsanity.com/paper/2307.00183/) (2023)
- [An Integrated System for Mobile Image-Based Dietary Assessment](https://mlsanity.com/paper/2110.01754/) (2021)
- [Long-tailed Food Classification](https://mlsanity.com/paper/2210.14748/) (2022)
- [Exemplar-free Online Continual Learning](https://mlsanity.com/paper/2202.05491/) (2022)
- [Online Class-Incremental Learning For Real-World Food Image Classification](https://mlsanity.com/paper/2301.05246/) (2023)
- [Improving Dietary Assessment Via Integrated Hierarchy Food Classification](https://mlsanity.com/paper/2109.02736/) (2021)
- [An End-to-end Food Portion Estimation Framework Based on Shape Reconstruction from Monocular Image](https://mlsanity.com/paper/2308.01810/) (2023)
- [Single-Stage Heavy-Tailed Food Classification](https://mlsanity.com/paper/2307.00182/) (2023)
- [Personalized Classifier for Food Image Recognition](https://mlsanity.com/paper/1804.04600/) (2018)
- [Simulating Personal Food Consumption Patterns using a Modified Markov Chain](https://mlsanity.com/paper/2208.06709/) (2022)
- [Conditional Synthetic Food Image Generation](https://mlsanity.com/paper/2303.09005/) (2023)
- [Hierarchical Image Classification with A Literally Toy Dataset](https://mlsanity.com/paper/2111.00892/) (2021)
- [Smartphone Apps for Tracking Food Consumption and Recommendations: Evaluating Artificial Intelligence-based Functionalities, Features and Quality of Current Apps](https://mlsanity.com/paper/2208.02490/) (2022)
- [Self-Supervised Visual Representation Learning on Food Images](https://mlsanity.com/paper/2303.09046/) (2023)

## Built on by (most cited first, 4 of 4)

- [MFP3D: Monocular Food Portion Estimation Leveraging 3D Point Clouds](https://mlsanity.com/paper/2411.10492/) (2024)
- [Personalized Food Image Classification: Benchmark Datasets and New Baseline](https://mlsanity.com/paper/2309.08744/) (2023)
- [Comprehensive Evaluation of Large Multimodal Models for Nutrition Analysis: A New Benchmark Enriched with Contextual Metadata](https://mlsanity.com/paper/2507.07048/) (2025)
- [FMiFood: Multi-modal Contrastive Learning for Food Image Classification](https://mlsanity.com/paper/2408.03922/) (2024)

## Abstract (arXiv)

Food image classification serves as a fundamental and critical step in image-based dietary assessment, facilitating nutrient intake analysis from captured food images. However, existing works in food classification predominantly focuses on predicting 'food types', which do not contain direct nutritional composition information. This limitation arises from the inherent discrepancies in nutrition databases, which are tasked with associating each 'food item' with its respective information. Therefore, in this work we aim to classify food items to align with nutrition database. To this end, we first introduce VFN-nutrient dataset by annotating each food image in VFN with a food item that includes nutritional composition information. Such annotation of food items, being more discriminative than food types, creates a hierarchical structure within the dataset. However, since the food item annotations are solely based on nutritional composition information, they do not always show visual relations with each other, which poses significant challenges when applying deep learning-based techniques for classification. To address this issue, we then propose a multi-stage hierarchical framework for food item classification by iteratively clustering and merging food items during the training process, which allows the deep model to extract image features that are discriminative across labels. Our method is evaluated on VFN-nutrient dataset and achieve promising results compared with existing work in terms of both food type and food item classification.