# Multimodal ML: Quantifying the Improvement of Calorie Estimation Through Image-Text Pairs

arXiv: https://arxiv.org/abs/2511.11705 · HTML: https://mlsanity.com/paper/2511.11705/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Builds on (most cited first, 6 of 6)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Multi-Task Learning for Calorie Prediction on a Novel Large-Scale Recipe Dataset Enriched with Nutritional Information](https://mlsanity.com/paper/2011.01082/) (2020)
- [An Improved Encoder-Decoder Framework for Food Energy Estimation](https://mlsanity.com/paper/2309.00468/) (2023)
- [CaLoRAify: Calorie Estimation with Visual-Text Pairing and LoRA-Driven Visual Language Models](https://mlsanity.com/paper/2412.09936/) (2024)

## Abstract (arXiv)

This paper determines the extent to which short textual inputs (in this case, names of dishes) can improve calorie estimation compared to an image-only baseline model and whether any improvements are statistically significant. Utilizes the TensorFlow library and the Nutrition5k dataset (curated by Google) to train both an image-only CNN and multimodal CNN that accepts both text and an image as input. The MAE of calorie estimations was reduced by 1.06 kcal from 84.76 kcal to 83.70 kcal (1.25% improvement) when using the multimodal model.