# Calorie Aware Automatic Meal Kit Generation from an Image

arXiv: https://arxiv.org/abs/2112.09839 · HTML: https://mlsanity.com/paper/2112.09839/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 18 Dec 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [CNN-RNN: A Unified Framework for Multi-label Image Classification](https://mlsanity.com/paper/1604.04573/) (2016)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [Boosting Image Captioning with Attributes](https://mlsanity.com/paper/1611.01646/) (2016)
- [Inverse Cooking: Recipe Generation from Food Images](https://mlsanity.com/paper/1812.06164/) (2018)
- [Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings](https://mlsanity.com/paper/1804.11146/) (2018)
- [Captioning Images with Diverse Objects](https://mlsanity.com/paper/1606.07770/) (2016)
- [Functional Object-Oriented Network for Manipulation Learning](https://mlsanity.com/paper/1902.01537/) (2019)
- [Learning Deep Latent Spaces for Multi-Label Classification](https://mlsanity.com/paper/1707.00418/) (2017)
- [Wide-Slice Residual Networks for Food Recognition](https://mlsanity.com/paper/1612.06543/) (2016)
- [Functional Object-Oriented Network: Construction & Expansion](https://mlsanity.com/paper/1807.02189/) (2018)
- [Joint Object and State Recognition using Language Knowledge](https://mlsanity.com/paper/1905.08843/) (2019)
- [Personalized Classifier for Food Image Recognition](https://mlsanity.com/paper/1804.04600/) (2018)
- [Long Activity Video Understanding using Functional Object-Oriented Network](https://mlsanity.com/paper/1807.00983/) (2018)
- [Identifying Object States in Cooking-Related Images](https://mlsanity.com/paper/1805.06956/) (2018)
- [Evaluating Recipes Generated from Functional Object-Oriented Network](https://mlsanity.com/paper/2106.00728/) (2021)
- [Estimating Motion Codes from Demonstration Videos](https://mlsanity.com/paper/2007.15841/) (2020)
- [Developing Motion Code Embedding for Action Recognition in Videos](https://mlsanity.com/paper/2012.05438/) (2020)
- [Functional Task Tree Generation from a Knowledge Graph to Solve Unseen Problems](https://mlsanity.com/paper/2112.02433/) (2021)

## Abstract (arXiv)

Calorie and nutrition research has attained increased interest in recent years. But, due to the complexity of the problem, literature in this area focuses on a limited subset of ingredients or dish types and simple convolutional neural networks or traditional machine learning. Simultaneously, estimation of ingredient portions can help improve calorie estimation and meal re-production from a given image. In this paper, given a single cooking image, a pipeline for calorie estimation and meal re-production for different servings of the meal is proposed. The pipeline contains two stages. In the first stage, a set of ingredients associated with the meal in the given image are predicted. In the second stage, given image features and ingredients, portions of the ingredients and finally the total meal calorie are simultaneously estimated using a deep transformer-based model. Portion estimation introduced in the model helps improve calorie estimation and is also beneficial for meal re-production in different serving sizes. To demonstrate the benefits of the pipeline, the model can be used for meal kits generation. To evaluate the pipeline, the large scale dataset Recipe1M is used. Prior to experiments, the Recipe1M dataset is parsed and explicitly annotated with portions of ingredients. Experiments show that using ingredients and their portions significantly improves calorie estimation. Also, a visual interface is created in which a user can interact with the pipeline to reach accurate calorie estimations and generate a meal kit for cooking purposes.