# Nutrition Estimation for Dietary Management: A Transformer Approach with Depth Sensing

arXiv: https://arxiv.org/abs/2406.01938 · HTML: https://mlsanity.com/paper/2406.01938/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Libra R-CNN: Towards Balanced Learning for Object Detection](https://mlsanity.com/paper/1904.02701/) (2019)
- [Visual Saliency Transformer](https://mlsanity.com/paper/2104.12099/) (2021)
- [Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food](https://mlsanity.com/paper/2103.03375/) (2021)
- [Image Based Food Energy Estimation With Depth Domain Adaptation](https://mlsanity.com/paper/2208.12153/) (2022)
- [From Canteen Food to Daily Meals: Generalizing Food Recognition to More Practical Scenarios](https://mlsanity.com/paper/2403.07403/) (2024)
- [A Comprehensive Survey of Image-Based Food Recognition and Volume Estimation Methods for Dietary Assessment](https://mlsanity.com/paper/2106.11776/) (2021)
- [NutritionVerse-Synth: An Open Access Synthetically Generated 2D Food Scene Dataset for Dietary Intake Estimation](https://mlsanity.com/paper/2312.06192/) (2023)

## Built on by (most cited first, 4 of 4)

- [GiNet: Integrating Sequential and Context-Aware Learning for Battery Capacity Prediction](https://mlsanity.com/paper/2501.04997/) (2025)
- [ReLA: Representation Learning and Aggregation for Job Scheduling with Reinforcement Learning](https://mlsanity.com/paper/2601.03646/) (2026)
- [Not Your Stereo-Typical Estimator: Combining Vision and Language for Volume Perception](https://mlsanity.com/paper/2604.09886/) (2026)
- [NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation](https://mlsanity.com/paper/2609.33076/) (2026)

## Abstract (arXiv)

Nutrition estimation is crucial for effective dietary management and overall health and well-being. Existing methods often struggle with sub-optimal accuracy and can be time-consuming. In this paper, we propose NuNet, a transformer-based network designed for nutrition estimation that utilizes both RGB and depth information from food images. We have designed and implemented a multi-scale encoder and decoder, along with two types of feature fusion modules, specialized for estimating five nutritional factors. These modules effectively balance the efficiency and effectiveness of feature extraction with flexible usage of our customized attention mechanisms and fusion strategies. Our experimental study shows that NuNet outperforms its variants and existing solutions significantly for nutrition estimation. It achieves an error rate of 15.65%, the lowest known to us, largely due to our multi-scale architecture and fusion modules. This research holds practical values for dietary management with huge potential for transnational research and deployment and could inspire other applications involving multiple data types with varying degrees of importance.