# Multi-Task Image-Based Dietary Assessment for Food Recognition and Portion Size Estimation

arXiv: https://arxiv.org/abs/2004.13188 · HTML: https://mlsanity.com/paper/2004.13188/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2020.

## Built on by (most cited first, 26 of 26)

- [Online Continual Learning For Visual Food Classification](https://mlsanity.com/paper/2108.06781/) (2021)
- [An End-to-End Food Image Analysis System](https://mlsanity.com/paper/2102.00645/) (2021)
- [Long-Tailed Continual Learning For Visual Food Recognition](https://mlsanity.com/paper/2307.00183/) (2023)
- [An Integrated System for Mobile Image-Based Dietary Assessment](https://mlsanity.com/paper/2110.01754/) (2021)
- [Long-tailed Food Classification](https://mlsanity.com/paper/2210.14748/) (2022)
- [Improving Dietary Assessment Via Integrated Hierarchy Food Classification](https://mlsanity.com/paper/2109.02736/) (2021)
- [Food Portion Estimation via 3D Object Scaling](https://mlsanity.com/paper/2404.12257/) (2024)
- [Single-Stage Heavy-Tailed Food Classification](https://mlsanity.com/paper/2307.00182/) (2023)
- [A review on vision-based analysis for automatic dietary assessment](https://mlsanity.com/paper/2108.02947/) (2021)
- [MetaFood CVPR 2024 Challenge on Physically Informed 3D Food Reconstruction: Methods and Results](https://mlsanity.com/paper/2407.09285/) (2024)
- [MFP3D: Monocular Food Portion Estimation Leveraging 3D Point Clouds](https://mlsanity.com/paper/2411.10492/) (2024)
- [A Comprehensive Survey of Image-Based Food Recognition and Volume Estimation Methods for Dietary Assessment](https://mlsanity.com/paper/2106.11776/) (2021)
- [Simulating Personal Food Consumption Patterns using a Modified Markov Chain](https://mlsanity.com/paper/2208.06709/) (2022)
- [An Improved Encoder-Decoder Framework for Food Energy Estimation](https://mlsanity.com/paper/2309.00468/) (2023)
- [Towards the Creation of a Nutrition and Food Group Based Image Database](https://mlsanity.com/paper/2206.02086/) (2022)
- [Diffusion Model with Clustering-based Conditioning for Food Image Generation](https://mlsanity.com/paper/2309.00199/) (2023)
- [Muti-Stage Hierarchical Food Classification](https://mlsanity.com/paper/2309.01075/) (2023)
- [Personalized Food Image Classification: Benchmark Datasets and New Baseline](https://mlsanity.com/paper/2309.08744/) (2023)
- [Automatic Recognition of Food Ingestion Environment from the AIM-2 Wearable Sensor](https://mlsanity.com/paper/2405.07827/) (2024)
- [Conditional Synthetic Food Image Generation](https://mlsanity.com/paper/2303.09005/) (2023)
- [Synthetic Data Augmentation using Pre-trained Diffusion Models for Long-tailed Food Image Classification](https://mlsanity.com/paper/2506.01368/) (2025)
- [Comprehensive Evaluation of Large Multimodal Models for Nutrition Analysis: A New Benchmark Enriched with Contextual Metadata](https://mlsanity.com/paper/2507.07048/) (2025)
- [Self-Supervised Visual Representation Learning on Food Images](https://mlsanity.com/paper/2303.09046/) (2023)
- [SalienTrack: providing salient information for semi-automated self-tracking feedback with model explanations](https://mlsanity.com/paper/2109.10231/) (2021)
- [V-Nutri: Dish-Level Nutrition Estimation from Egocentric Cooking Videos](https://mlsanity.com/paper/2604.11913/) (2026)
- [Geometry-Enhanced Portion Estimation for Multimodal LLMs](https://mlsanity.com/paper/2607.16514/) (2026)

## Abstract (arXiv)

Deep learning based methods have achieved impressive results in many applications for image-based diet assessment such as food classification and food portion size estimation. However, existing methods only focus on one task at a time, making it difficult to apply in real life when multiple tasks need to be processed together. In this work, we propose an end-to-end multi-task framework that can achieve both food classification and food portion size estimation. We introduce a food image dataset collected from a nutrition study where the groundtruth food portion is provided by registered dietitians. The multi-task learning uses L2-norm based soft parameter sharing to train the classification and regression tasks simultaneously. We also propose the use of cross-domain feature adaptation together with normalization to further improve the performance of food portion size estimation. Our results outperforms the baseline methods for both classification accuracy and mean absolute error for portion estimation, which shows great potential for advancing the field of image-based dietary assessment.