# Multi-Task Learning for Calorie Prediction on a Novel Large-Scale Recipe Dataset Enriched with Nutritional Information

arXiv: https://arxiv.org/abs/2011.01082 · HTML: https://mlsanity.com/paper/2011.01082/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2020.

## Built on by (most cited first, 8 of 8)

- [Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food](https://mlsanity.com/paper/2103.03375/) (2021)
- [NutriBench: A Dataset for Evaluating Large Language Models on Nutrition Estimation from Meal Descriptions](https://mlsanity.com/paper/2407.12843/) (2024)
- [A review on vision-based analysis for automatic dietary assessment](https://mlsanity.com/paper/2108.02947/) (2021)
- [Should I take a walk? Estimating Energy Expenditure from Video Data](https://mlsanity.com/paper/2202.00712/) (2022)
- [SFOOD: A Multimodal Benchmark for Comprehensive Food Attribute Analysis Beyond RGB with Spectral Insights](https://mlsanity.com/paper/2507.04412/) (2025)
- [Investigating the Impact of Large-Scale Pre-training on Nutritional Content Estimation from 2D Images](https://mlsanity.com/paper/2508.03996/) (2025)
- [Multimodal ML: Quantifying the Improvement of Calorie Estimation Through Image-Text Pairs](https://mlsanity.com/paper/2511.11705/) (2025)
- [Doppio: A Dataset for Contactless Weight Estimation of Falling Particles](https://mlsanity.com/paper/2609.02528/) (2026)

## Abstract (arXiv)

A rapidly growing amount of content posted online, such as food recipes, opens doors to new exciting applications at the intersection of vision and language. In this work, we aim to estimate the calorie amount of a meal directly from an image by learning from recipes people have published on the Internet, thus skipping time-consuming manual data annotation. Since there are few large-scale publicly available datasets captured in unconstrained environments, we propose the pic2kcal benchmark comprising 308,000 images from over 70,000 recipes including photographs, ingredients and instructions. To obtain nutritional information of the ingredients and automatically determine the ground-truth calorie value, we match the items in the recipes with structured information from a food item database. We evaluate various neural networks for regression of the calorie quantity and extend them with the multi-task paradigm. Our learning procedure combines the calorie estimation with prediction of proteins, carbohydrates, and fat amounts as well as a multi-label ingredient classification. Our experiments demonstrate clear benefits of multi-task learning for calorie estimation, surpassing the single-task calorie regression by 9.9%. To encourage further research on this task, we make the code for generating the dataset and the models publicly available.